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EXECUTIVE  SUMMARY 


The  basic  purpose  of  this  study  was  to  investigate  the  guidance 
and  rationale  behind  current  USAF  test  and  evaluation  policy  found  in  APR 
80-14,  Test  and  Evaluation,  and  AFR  23-36,  Air  Force  Test  and  Evaluation 
Center  (AFTBC) ; to  make  reocxiitiendations  to  the  program  manager  concerning 
the  use  of  these  regulations;  and  finally  to  provide  a source  document  of 
the  corporate  memory  involved  in  the  revision  of  these  regulations. 

To  assess  current  T&E  policy,  a review  was  first  made  of  corres- 
pondence and  direction  resulting  in  the  initiation  of  the  regulations' 
revision.  Prior  versions  of  the  regulations  and  Department  of  Defense 
Directives  (DcCO)  were  coitpared  with  the  current  ones  to  determine  re- 
sultant policy  changes  and  trends,  /^ropriate  journal  articles  and  other 
AF  regulations  were  reviewed,  and  finally,  discussions  with  Air  Staff  per- 
sonnel that  participated  in  the  revision  of  the  regulations  were  held  to 
gain  insight  into  rationale  for  the  policy  changes.  The  r^xart  concludes 
with  reoormendations  to  the  program  manager  as  to  how  to  make  better  use 
of  the  T&E  regulations. 

Ihe  significant  results  include  a complete  discussion  of  the  rea- 
soning behind  the  major  aspects  of  USAF  T&E  policy  and  a review  of  the 
draft  DoDD  5000.3,  Test  and  Evalxoation,  as  it  might  inpact  AF  policy. 
Detailed  review  of  DoDD  5000. 3 showed  that  there  ette  several  proposed 
changes  in  policy.  Separate  testing  is  no  longer  stressed  but  rather 

11 


coTtoined  test  programs  when  cost/time  benefits  resxalt.  However,  the  draft 
proposes  that  the  OT&E  agency  independently  conduct  operational  tests  sup- 
porting production  decision.  Iitportanoe  is  also  given  to  consideration  of 
and  planning  for,  incorporation  of  software  into  the  aoguisition  process. 
Another  new  requirenent  is  the  coordination  and  review  of  the  Test  and 
Evaluation  Master  Plan  (TE!'®*)  by  OSD.  This  proposal  spears  to  conflict 
with  DoEO  5000.1  direction  to  decentralize  responsibility  for  systan  ac- 
quisition managanent. 

This  stucfy  will  be  useful  to  not  only  program  managers  but  also  to 
program  office  personnel  and  DT&E  and  OTSE  test  team  manbers  in  gaining 
fuller  understanding  of  their  role  in  T&E.  Moreover,  it  provides  a 
corporate  memory  that  is  not  recorded  elsevdiere  which  can  be  used  by  HQ 
USAF  and  MAJCOM  personnel  involved  in  T&E  policy  and  also  historical  re- 
searchers. 


ill 


ACKNOWLEDGEMENT 


I wish  to  acknowledge  the  support,  perseverance,  and  dedication 
from  Mr.  Howard  A.  Beck  of  USAF/XOODE  for  the  many  months  of  work 
with  the  author  in  revising  AFRs  80-14  and  23-36,  and  also  his 
review  and  assistance  in  preparing  this  report. 


iv 


TABLE  OF  CONTENTS 


EXECUTIVE  SUMMARY  ii 

ACKNOWLEDGEMENT  iv 

I.  PURPOSE,  GOALS  AND  SCOPE  1 

II.  BACKGROUND 3 

III.  CURRENT  USAF  T&E  PHILOSOPHY  & POLICY 8 

General  Comments  8 

Types  of  OT&E 12 

A New  Definition  for  lOT&E 12 

AFTEC's  Role  in  FOT&E 14 

Qualification/Operational  Test  and  Evaluation  (OT&E)  . . 15 

Relationship  Between  DT&E  and  0T6tE 16 

General  Statement  of  T&E  Management  Policy  17 

OT&E  Test  Program  Management 18 

Innovative  0T6tE  Management  Concepts 20 

Responsibilities  of  the  Test  Director 20 

Relationship  Between  the  OT&E  Test  Director  and  the 

Test  Director 21 

Misconceptions  of  Combined  Testing  22 

OT&E  in  Modification  Programs 23 

Program  Reviews  24 

Operating  and  Support  (O&S)  Costs  24 

Foreign  Military  Sales  (FMS)  24 

IV.  EFFECTS  OF  NEW  DOD  DIRECTION  PERTAINING  TO  USAF 

T&E  POLICY 29 

New  Guidance 29 

Revision  of  DoD  Directives 29 

DT&E  No  Longer  to  Assess  Military  Utility? 30 

AFTEC  to  Participate  in  Planning  of  DT&E? 32 

Some  Changes  Led  by  the  Air  Force 32 

Combined  Versus  Independent  Testing  33 

Long- Lead  Items  34 

TEMP 34 

Software  Emphasized  35 

Flexible  Management  35 

V.  SUMMARY  AND  RECOMMENDATIONS 37 

APPENDIX  A;  AFR  80-14,  19  July  1976 

APPENDIX  B:  AFR  23-36,  19  July  1976 


BIBLIOGRAPHY 


V 


SECTION  I 


PURPOSE,  GOALS  AND  SCOPE 

Ihe  test  and  evalviation  (T&E)  policy  of  the  US  Air  Force  is  em- 
bodied in  primarily  two  regulations:  AFR  80-14,  Test  and  Evaluation,  and 
AFR  23-36,  Air  Force  Test  and  Evaluation  Center  (APTBC) . Both  were  re- 
vised in  1976  in  response  to  direct  guidance  by  the  USAF  Chief  of  Staff 
to  bring  greater  clarity  to  T&E  policy  and  to  modify  the  APTBC  charter. 

Purpose 

The  purpose  of  this  study  is  to  investigate  the  guidance  and  ra- 
tionale behind  these  revisions  with  the  ultimate  goal  of  increasing  man- 
agement's understanding  of  its  role  in  test  and  evaluation.  There  is 
often  a tendency  to  translate  a regulation  too  literally  without  under- 
standing the  real  purpose  behind  a certain  policy.  This  study  clarifies 
the  reasoning  behind  current  T&E  policy  in  hopes  of  preventing  misinter- 
pretations in  the  future. 

An  additional  purpose  of  the  stut^  is  to  provide  a documented 
accounting  of  the  corporate  itanory  involved  in  rewriting  these  regula- 
tions. Since  most  of  the  individuals  involved  have  conpleted  their 
assigiments  at  the  Pentagon,  this  report  should  be  a valuable  resource 
document  for  new  personnel  that  might  become  involved  in  future  revisions 
of  the  regulations. 


Goals  and  Scope 

To  assess  cijrrent  T&E  policy,  a review  was  first  made  of  letters 
and  direction  that  resulted  in  the  initiation  of  the  regulations'  re- 
visicn.  Prior  versions  of  the  regulations  and  pertinent  Department  of 
Etefense  Directives  (DoED)  were  ccitpared  with  the  current  ones  to  determine 
resulting  policy  changes  and  trends.  In  addition,  appropriate  journal 
articles  and  other  Air  Force  regulations  were  reviewed  for  their  iitpli- 
cations  in  T&E  policy.  Finally,  discussions  were  held  with  Air  Staff 
personnel  that  participated  in  the  revision  of  the  regulations  to  gain 
insight  into  the  rationale  behind  policy  cdianges.  In  addition  to  provid- 
ing a better  understanding  of  USAF  T&E  policy  and  hew  future  DoD  policy 
may  affect  it,  the  report  concludes  with  reconmendations  to  the  program 
manager  concerning  hew  to  make  better  use  of  the  T&E  regulations. 

The  scope  of  this  stuefy  is  limited  to  the  Air  Force  ccnponent;  to 
the  major  T&E  changes  in  ARFs  80-14  and  23-36  (dated  19  July  1976)  and 
DolXis  5000.1,  fiajor  Systatis  Acquisitions,  and  5000.2,  Major  System  Ac- 
cjuisition  Prexoss  (dated  18  January  1977) ; and  also  to  major  changes  in 
the  draft  DoEO  5000.3,  Test  and  Evaluation,  which  is  still  under  revision. 
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SECTION  II 


B^CRSRUUMD 

In  mid  1975,  two  letters  were  primary  in  triggering  the  revision 
of  APR  80-14,  and  AFR  23-36.  The  first  was  from  Lt  General  Hails,  Vice 
Ccmnander  TAC,  to  Lt  General  Huyser,  Deputy  CMef  of  Staff,  Plans  and 
Operations,  Hq  USAF  (1:1).^  In  his  letter.  General  Hails  stressed  full 
support  by  TAC  for  AFTEC's  role,  but  questioned  the  need  for  AFTEC  to 
manage  cperational  T&E  (OT&E)  after  the  production  decision.  He  felt  the 
user  could  best  conduct  the  follcw-on  testing  (FCT&E) . 

The  second  letter  was  from  AFTBC's  Commander,  Major  General  Cross, 
to  the  USAF  Chief  of  Staff  (CSAF) , (2) . Since  General  Cross  was  about  to 
retire,  he  felt  it  appropriate  to  give  his  iitpressions  of  AFTEC  since  its 
establishment.  He  was  particularly  worried  that  the  General  Accounting 
Office  (GAD) , which  at  the  time  was  doing  a special  survey  of  the  Ser- 
vices' OT&E  organizations  (5:2),  would  recomnend  that  OSD  establish  an 
independent  OSD  test  organization,  as  GAD  did  not  believe  that  AFTEC,  OTEA 
and  OPTEVDR  had  achieved  their  independence.  In  this  regard,  he  reootrmended 
these  vital  actions  (2) : 

• Clearly  establish  the  independence  of  AFTEC 's  missions. 

• Revise  AFR  80-14  to: 

4liis  notation  will  be  used  throughout  the  report  for  sources  of 
quotations  and  major  references.  The  first  nunber  is  the  source  listed  in 
the  bibliography.  The  second  nunber  is  the  page  in  the  reference. 
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1.  Eliminate  diiplication  of  testing  responsibilities. 

2.  Reassess  the  cJelegation  of  responsibility  for  overall 
management  of  DT&E  and  initial  operational  test  and 
evaluation  (lOT&E) . 

3.  Assure  dedicated  lOT&E  at  the  earliest  possible  time 
in  DT&E/IOT&E  test  programs. 

4.  Establish  organization  of  OT&E  teams  both  in  lOT&E 
and  POT&E  and  outline  relationships  with  MAJCX^IS  and 
the  inplanenting  ccnmand. 

• Provide  APTBC  the  flexibility  to  acootplish  OT&E  in  the 

most  realistic  environment. 

It  had  been  a year  and  a half  since  the  establishment  of  AFTBC.  As 
with  any  new  organization,  APTBC  had  experienced  growing  pains  and  had  had 
to  feel  out  and  establish  its  cwn  place  in  the  test  ccmrtunity.  But  now 
was  a time  for  reevaluation  and  redefinition  of  its  role  in  Air  Force  ac- 
quisition programs. 

Before  proceeding  any  further,  it  should  be  noted  that  when  regula- 
tions are  revised  at  Headquarters  USAF,  coordination  generally  involves 
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Directorate  level  or  belcw.  The  Air  Staff  office  of  primary  responsibility 
(OPR)  coordinates  the  revision  with  other  appropriate  Air  Staff  offices 
and  major  ocnmands.  However,  in  this  case,  the  Chief  of  Staff  felt  this 

^Directorates  are  the  next  organizational  level  below  Deputy  Chiefs 
of  Staff  (See  Chart  1). 
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All  STAFF 


CHART  1.  ORGANIZATION  OF  THE  AIR  STAFF 


issue  was  irrportant  enough  that  he  personally  gave  sane  very  specific 
guidance  concerning  AFTEC's  role  to  be  incorporated  into  the  aj^licable 
regulations . 

General  Jones  provided  the  following  thoughts  to  the  Air  Staff 
concerning  the  TAG  and  ATTEC  issues: 

• He  wanted  AFTBC  to  have  more  clout  and  conplete  control  of 
lOT&E. 

• AFTBC  was  to  assume  an  adversary  role  as  the  independent 
test  agency. 

• He  wanted  AFTBC  personnel  in  the  plant  and  the  System 
Program  Office  (SPO)  to  learn  as  much  as  possible  about 
the  new  system. 

• AFTBC  should  be  involved  in  fewer  programs,  but  do  them 
bettor. 

To  acoonplish  this  direction,  the  Air  Staff  inmediately  established 
a T&E  Policy  Working  Gixxp  made-vp  of  members  from  across  the  Air  Staff 
(4).  The  group  was  co-chaired  by  the  Chiefs  of  the  OT&E  Di''Hsion  (formally 
AF/XOOFA,  new  AF/XOCDE)  and  the  Assistant  for  Acquisition  Management 
Office  (formally  AF/RDM,  now  the  Test  and  Evaluation  Division:  AF/RDXT) 
v\rtiose  offices  have  primary  responsibility  (OPR)  for  APR  23-36  and  APR 
80-14  respectively.  One  of  their  first  actions  was  to  solicit  iiputs 
from  all  affected  MAJCOMS  and  Air  Staff  agencies  (3) . 

While  awaiting  responses,  the  groip  developed  their  revision 
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strategy.  Both  regulations  would  be  revised  concurrently  to  insure  con- 
tinuity. Thou<^  some  of  the  MAJCXM  reo>jnmendations  varied  widely  from 
establishing  policy,  the  Wbrking  Groi;p  decided  not  to  be  revolutionary, 
but  rather  to  be  evolutionary,  staying  within  the  present  policy  frame- 
work to  clarify  and  eitphasize  respective  responsibilities.  Guidance  from  the 
the  Chief  of  Staff  and  direction  foiand  in  DoD  Directive  5000.3,  Test  and 
Evaluation,  would  be  used  as  baselines. 

In  addition,  other  issues  needed  resolution  (26): 

• Soope  of  AFTEC  involvement  in  cost  of  ownership. 

• OT&E  gap  between  production  decision  and  availability  of 
production  hardware. 

• OT&E  in  modification  programs,  simulator  programs  and 
foreign  military  sales  (FMS) . 

• Responsibilities  in  combined  DT&E/OT&E  programs. 

• Control  over  CT&E  test  teams. 

• AFTEC  versus  MAJCOM  responsibility  for  conduct  of  FOT&E. 
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SECTION  III 


CURRiyiT  USAF  T&E  PHILOSOPHY  & POLICY 

GBSIERAL  OOMMEOTS 

Through  the  fall  of  1975,  the  T&E  Policy  Working  Groxp  labored 
over  iteration  and  reiteration  of  drafts  to  AFRs  80-14  and  23-36.  Grad- 
ually, a policy  was  developed  within  both  the  framework  of  existing  ctr- 
jectives  and  the  guidelines  set  forth  hy  the  Chief  of  Staff.  Where  no 
clear-cut  guidance  was  specified,  the  Group  based  decisions  on  their  in- 
terpretation of  what  Congress'  intent  was  in  establishing  independent 
test  agencies.  In  this  manner,  the  Groi;p  hcped  to  follow  not  only  Con- 
gressional direction  but  the  spirit  and  intent  of  that  direction  so  that 
the  Air  Force  would  be  above  criticism  in  its  conduct  of  test  and  evalua- 
tion. 

Related  to  the  concern  of  following  the  latest  policy  guidance  was 
the  Working  Grotp's  desire  to  keep  the  regulations  as  general  as  possible 
(6).  Terms  such  as  "normally,"  "usually"  or  "generally"  were  often  used. 
The  rationale  for  this  approach  was  not  for  the  purpose  of  appearing 
vague  or  providing  inadequate  direction.  The ''generality"  was  incorporated 
because  there  are  no  cook-book  methods  for  managing  either  a system  ac- 
quisition program  or  the  test  portion  of  a program.  As  a result,  the 
regulations  were  revised  to  give  program  managers  and  test  directors  the 
widest  latitude  in  managing  their  programs  vhile  staying  within  the 
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bomds  of  the  aforementioned  guidance. 

Another  philosophy  that  was  basic  throughout  the  revision  was  to 
assure  that  vdiatever  was  written  would  be  applicable  to  all  systems,  be 
it  a light  bulb  or  a giant  rocket,  and  not  just  aircraft. 

Since  all  policy  changes  were  first  addressed  in  AFR  80-14  and  the 
changes  that  apply  to  AFR  23-36  were  later  added  for  consistency  purposes, 
the  follov/ing  dissertation  concerns  only  AFR  80-14  unless  otherwise  noted. 
When  the  term  "the  regulation"  is  used,  it  means  AFR  80-14. 

A primary  issue  that  needed  to  be  resolved  in  the  regulation  was 
"Just  what  ^ test  and  evaluation  during  the  acquisition  process?"  It 
appeared  that  over  the  year  and  a half  since  AFTEC's  inception,  many 
people  had  come  to  believe  there  were  three  types  of  T&E:  developmental 
(DT&E) , initial  (lOT&E) , and  follow-on  (POT&E) . This  assurrption  was 
probably  due  to  the  fact  that  the  earlier  version  of  AFR  80-14  appeared  to 
stress  the  breakout  of  testing  into  these  three  categories  (16:4-6). 

It  was  a common  assunption  by  many  that  all  programs  had  to  include  the 
three  types  during  the  testing  phase.  HCTvever,  this  assunption  often  led 
to  confusion  as  to  what  to  call  the  operational  testing  in  programs  that 
had  no  specific  requirement  for  DTSE,  such  as  certain  modification  pro- 
grams or  off-the-shelf  hardware.  As  a baseline,  the  Group  wanted  to  de- 
velop a clear  definition  of  what  T&E  is  comprised  of  and  specific  de- 
finitions for  the  various  divisions  of  T&E.  This  baseline  was  necessary 
before  any  of  the  other  issues  could  be  addressed. 
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The  Groip's  efforts  resulted  in  a strong  philosophy  concerning  T&E, 
and  they  felt  it  should  be  CTphasized  in  the  revision.  Hie  current  re- 
gulation, paragraph  2a,  states  specifically: 

"There  are  two  basic  kinds  of  test  and  evaluation: 

Developmental  Test  and  Evaluation  (OT&E)  and  Operational 

Test  and  Evaluation  (OT&E)"  (17:1). 

Thus,  whenever  T&E  is  used  in  the  regulation,  it  is  meant  to  include  both 
ET&E  and  OT&E.  Though  this  may  appear  as  a somevtiat  ctovious  statement,  it 
has  ijiportant  inplication.  For  instance,  paragraph  2a  goes  on  to  say: 

"The  primary  purpose  of  all  T&E  in  the  acquisition  process 

is  to  estimate  the  military  utility  of  a system " 

This  statement  indicates  the  cannon  thread  that  runs  through  all  testing, 
whether  developmental  or  operational:  T&E  is  to  assess  military  utility. 

An  exaitple  or  two  might  prove  helpful:  Exanple  1 - Though  developmental 
missions  are  very  precise  and  are  flown  under  controlled  conditions  to 
determine  and  verify  stability  and  control  or  performance  characteristics 
of  an  aircraft,  the  developmental  test  pilot  can  also  provide  valuable 
cissessments  to  the  operational  test  team  concerning  military  utility.  In 
fact,  an  exceptional  operational  background  is  a prime  requisite  for  selec- 
tion to  the  USAF  Test  Pilot  School.^  Thus,  the  Air  Force  Systems  Cermand 
(AFSC)  test  pilot  is  often  highly  qualified  to  recognize  a system's  opera- 
tional qualities  and  the  inpact  on  military  utility  that  might  beceme 

historically^  the  applicants  chosen  for  the  USAF  Test  Pilot  School 
have  experience  in  more  than  one  operaticnal  aircraift  and  otherwise  have 
excelled  such  as  #1  in  pilot  training.  Top  Gun  Award,  Fighter  Wing  Stand- 
board  Examiner,  Vfe^xDns  School  Graduate,  etc. 
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evident  during  a DT&E  mission.  Feedback  of  this  sort  could  result  in 
iitproved  OT&E  missions  and  possibly  reduce  the  number  of  flights  reqxaired 
by  the  OT&E  team.  Exanple  2 - Another  exaitple  is  the  laboratory  engi- 
neer involved  in  a military  systan.  It  has  been  noted  that  some  experi- 
mental engineers  possess  a natural  tendency  to  seek  technical  perfection. 
Sane  design  engineers  will  tinker  and  fiddle  indefinitely  in  their  quest 
for  perfection  with  the  resulting  consequence  of  schedule  slippage  and 
cost  problems.  Though  he  works  in  highly  specialized  and  technical  areas, 
sonetimes  with  concepts  so  abstract  that  there  is  no  foreseeable  use,  the 
engineer  still  must  never  forget  that  his  primary  goal  is  military  utility. 
If  the  resulting  system  is  too  technical,  too  cotplex,  hav  can  it  be  of 
real  use  to  the  soldier  on  the  battlefield?  The  last  thing  an  engineer 
should  want  is  to  have  his  system  described  in  the  manner  of  a quote 
attributed  to  a British  naval  officer: 

"The  extreme  ingenuity  of  this  system  rather  blinds  one  to 

its  utter  uselessness"  (21:32). 

Though  the  testing  baseline  of  military  utility  overlaps  between 
DT&E  and  CT&E,  there  are  primary  concerns  that  are  different  between  the 
two.  DT&E  enconpasses  the  detailed  engineering  analysis  of  the  system's 
performance,  demonstrating  thiat  the  system  meets  certain  thresholds  and 
specifications.  OT&E,  on  the  other  hand,  views  military  utility  in  terms 
of  the  validated  military  requirement  and  the  operational  erployment  and 
naintenance  ooncepts  as  pijepared  by  the  major  conmands. 
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Several  of  the  replies  to  the  Working  Group's  request  for  revision 


inputs  reconmended  that  OT&E  be  broken  down  into  various  categories  in 
addition  to  the  lOT&E/FOT&E  phases.  However,  the  Working  Grov?)  felt  that 
any  further  division  would  serve  no  real  purpose  and  would  only  tend  to 
overburden  and  add  confusion  to  the  aoquisition  process  vdiich  was  already 
beleaguered  with  acronyms. 

A New  Definition  for  lOT&E 

Since  DoDD  5000.3  called  for  "an  initial  phase  of  operational  test- 
ing (lOT&E)"  to  be  acoOTplished  prior  to  the  production  decision  (13:4), 
the  Working  Grovp  chose  to  stay  with  this  definition,  and  more  inportantly, 
that  lOT&E  would  terminate  with  the  first  major  production  decision. 
Thereafter,  all  cperational  testing  would  be  called  follow-on  operational 
test  and  evaluation  (POT&E) . This  differed  f rcm  the  prior  version  of  AFR 
80-14  which  had  cdlcwed  lOTSiE  to  continue  past  the  production  decision  if 
all  the  lOT&E  objectives  had  not  been  met  (16:5). 

There  were  several  reasons  for  the  definite  cutoff  of  lOT&E  at  the 
Milestone  (M.S.)  Ill  review: 

• The  cutoff  forces  the  OT&E  agency  to  do  accurate  and  com- 
plete lOT&E  planning.  If  an  lOT&E  objective  is  inportant  enough  to  be 
required  for  the  production  decision,  then  it  is  inportant  enough  to  he 
accomplished  by  the  production  decision.  Otherwise,  it  is  not  a valid 
pre-producticMi  objective,  or  if  it  is  valid,  M.S.  Ill  should  be  slipped 
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until  the  objective  can  be  acccnplished. 

• The  definite  cutoff  limits  the  amount  of  developmental  funds 
spent  for  c^rational  testing.  Since  the  cost  of  lOT&E  is  extracted  from 
RDT&E  funds  (3600  appropriations) , terminating  lOT&E  at  the  prodixhzion 
decision  changes  funding  to  the  operation  and  maintenance  (OSM)  account 
(3400  apprcpriations) . This  changeover  results  in  an  additional  benefit. 

It  reduces  the  possibility  of  criticism  by  Congress  and  the  GAO  of  the 
amount  of  money  spent  on  weapon  system  development.  By  specifically  lim- 
iting lOT&E,  the  change  prevents  lOT&E  frcm  lingering  on  long  after  the 
production  decision,  which  in  one  case  dragged  out  over  18  months  and  re- 
quired in  excess  of  $2  million  for  just  one  portion  of  the  lOT&E  yet  to 
be  conducted. 

• Finally,  a definite  cutoff  at  the  Mi lestone  III  review  allows 
a more  orderly  changeover  of  lOT&E  to  POT&E  in  the  Planning,  Progranming 
and  Budgeting  System  (PPBS)  cycle.  Otherwise,  the  possibility  of  lOT&E 
continuing  past  M. S.  Ill  to  seme  nebulous  point  in  time  presents  quite  a 
problan  to  the  planners  and  budgeteers  and  also  opens  the  Service  to 
Congressional  critician  for  overruns  if  the  schedule  slips. 

It  might  be  added  that  the  l»torking  Group  defined  the  "first  major 
production  decision"  as  that  particular  review  vhere  the  decision  is  made 
to  buy  the  system  in  large  quantities  as  opposed  to  an  earlier  M.S.  Ill 
that  might  have  approved  only  a limited  bi;^  of  a few  articles.  It  is 
stated  in  the  AFR  80-14  glossary  as  the  decision  to  begin  production  of 
procurement- funded  end  items  intended  for  Service  deployment  (17:3). 
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AFTBC's  Role  in  KXr&E 


After  the  production  decision  at  M.S.  Ill,  operational  testing  is 
called  POT&E,  as  mentioned  earlier.  Fundamental  to  the  concept  of  FOT&E 
is  that  operational  testing  can  continue  throughout  the  life  of  a system, 
lasting  many  years.  However,  in  directing  the  Ccnponent  OT&E  independent 
agency  to  acccnplish  follow^n  phases  of  OT&E,  DoEO  5000.3  does  not  specify 
hew  long  the  agency  shoiiLd  remain  involved  (13:4). 

In  resolving  this  issue,  the  Working  Groip  reviewed  the  guidance 
described  earlier  in  Sec:tion  II  . To  summarize  briefly  here,  TAG  wanted 
to  assume  control  of  FOT&E  from  AFTBC  after  the  produortion  decision  and 
the  eSAF  favored  reducing  AFTEC's  involvement  in  programs.  These  two 
requests  provided  the  basis  for  the  Working  Grovp's  decision  to  limit 
AFTEC's  management  of  POT&E:  The  extent  of  AFTBC's  participation  is  now 
determined  by  HQ  USAF  (15:1). 

AFTEC's  principal  role  in  POT&E  is  that  of  furnishing  the  degree 
of  independent  OT&E  required  to  refine  initial  estimates  made  in  lOT&E. 

This  requirement  is  used  in  each  program  as  a criteria  by  USAF  in  de- 
termining the  amount  of  AFTBC  participation  in  POT&B.  It  was  concluded  as 
a general  rule  that  AFTBC  should  ranain  involved  through  production  of  at 
least  the  first  article.  The  Working  Grorp  felt  that  AFTEC's  conduct  of 
FCT&E  at  its  start  was  necessary  to: 

• Evaluate  deficiency  corrections. 

• Support  further  production  decisions;  and 
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• Evaluate  system  configuration  changes  after  the  production 
decision. 

Qualification/Operational  Test  and  Evaluation  (QOT&E) 

The  discussion  ip  to  this  point  has  been  building  to  a very  im- 
portant and  basic  pratu.se  that  was  used  throughout  the  revision  of  AFR 
80-14:  OT&E  is  only  split  into  the  components  of  lOT&E  and  POT&E  vten 
there  is  an  acccrpanying  developmental  (DT&E)  program  involving  3600 
money.  By  OSD  direction  (9:251.2)  lOTSeE  is  paid  from  DT&E  funds,  and 
therefore,  if  there  is  no  development  (3600)  money  in  a program,  it  fol- 
la,vs  that  any  operational  testing  that  might  be  required  cannot  be  called 
"lOT&E. " The  question  then  is,  "What  do  we  call  this  type  of  operational 
testing?" 

The  answer  originally  proposed  was  to  go  back  to  the  basic  defini- 
tion of  T&E  and  siitply  name  this  special  testing,  "OT&E."  (After  all,  the 
ctojectives  of  OT&E  do  not  change  with  the  prefix  in  front  of  OT&E. ) This 
titling  was  to  be  applied  to  such  examples  as: 

"those  Class  IV  and  V mcdifications,  simulators,  corm- 
ercially  developed  items,  and  other  items  vdrLch  may  require 
no  development,  per  se  and  therefore,  no  DT&E " (17:3). 

Note  the  key  words:  "those items  which. ...  require  no  development." 

It  also  should  be  emphasized  that  not  all  simulators  or  all  modifications 

are  included,  only  those  which  cure  not  involved  in  a development  program 

with  (3600  money) , but  still  must  be  tested  operationally.  For  instance, 

an  off-the-shelf  tape  recorder  works  fine  at  sea  level  but  how  does  it 

work  at  40,000  feet  in  a six  g turn. 
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However,  people  had  becane  so  ingrained  in  referring  to  OT&E  as 
"initial"  or  "follcw-on,"  that  during  cxDordination  of  the  draft  regulation, 
the  notation  of  just  plain  "OTSE"  confused  than.  To  prevent  any  further 
confusion,  the  term  "qualification  OT&E"  was  introduced  to  indicate  the 
need  for  operationally  qualifying  an  item  for  its  entry  into  the  inventory. 

Relationship  Between  DT&E  and  OT&E 

Upon  defining  T&E  and  it  conponents,  the  Vforking  Group  next  eval- 
uated the  relationship  among  these  coiponents.  One  of  the  most  inportant 
issues  discussed  was  the  need  for  separate  OT&E.  This  issue  was  closely 
related  to  the  USAF  Chief  of  Staff's  concern  over  the  amount  of  control 
that  AFTBC  had  in  conducting  lOT&E.  After  considerable  discussion  among 
the  Working  Groip  members,  it  was  concluded  that  separate  OT&E  was  the 
basic  intent  of  Congress  and  also  the  Blue  Ribbon  Defense  Panel  (20:9), 
even  though  in  actuality,  much  of  the  testing  is  done  in  a combined  de- 
velopment/operational test  environment.  To  follow  the  Chief's  direction 
of  more  clout  for  AFTEC  and  to  prevent  GAD  criticism  that  the  Air  Force 
regulation  differed  from  the  DoD  directive,  the  Working  Groxp  placed  the 
following  words  in  the  regulation  exactly  as  they  appear  in  EXoCt)  5000. 3 
(13:4): 

Operational  testing  should  be  separate  from  development  testing. 

Obviously  this  philosophy  is  an  ideal  one,  and  both  USAF  and  DcO 
acknowledge  this  by  allowing  combined  testing  vhere  separation  would 
cause  unacceptable  delay  or  increase  in  cost. 
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General  Statement  of  T&E  Management  Polii 


Air  Force  Regulation  800-2  (18:1)  states  that  "all  acquisition 

programs  shall  be  managed  by  a single  individual  known  as  the  Program 
Manager."  This  philosophy  was  incorporated  in  AFR  80-14,  but  with  a 
clarifying  exception: 

The  program  manager  has  overall  responsibility  for  a system 

acquisition  program  (except  the  management  of  OT&E).  (17:3) 

The  Working  Group  felt  that  excluding  the  management  of  OT&E  was  an  im- 
portant and  necessary  exception  in  order  to  give  AFTBC  more  control  of 
OT&E  and  also  to  show  the  Air  Force' s desire  to  endorse  AFTBC' s complete 
independence  from  the  develcping  ccittnand. 

However,  AFTEC  must  also  be  independent  from  the  using  and  sup- 
porting coitroands.  Once  the  OT&E  Test  Team  is  formed  of  personnel  from 
these  ccnmands,  the  OT&E  Test  Director  must  have  full  authority  if  he  is 
to  do  his  job  well.  As  a result,  the  regulation  now  directs  that  the 
OT&E  Test  Director  have  operational  control  over  the  OT&E  team  to  include 
decision  authority  over  the  lose  or  movement  of  all  assigned  OT&E  team 
members  and  OT&E  resources.  The  team  normally  is  oonprised  of  personnel 
from  at  least  several  ccmands,  such  as  TAC,  ATC,  and  AFLC. 

Regardless  of  the  broad  responsibilities  of  the  OT&E  Test  Director, 
the  individual  with  overall  responsibility  for  a program  is  the  program 
manager  who  has  the  job  of  "incorporating  the  OT&E  requironents  into  the 
test  program"  (17: 3)  in  addition  to  managing  OT&E.  Although  the  Test  Di- 
rector must  provide  support  for  OT&E  as  apprcpriate,  the  OT&E  Test 
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Director,  on  the  other  hand,  must  make  the  resources  \mder  his  operational 
control  available  to  support  the  DT&E  test  plan  when  specified  in  the  test 
documentation.  If  additional  DT&E  support  is  needed  from  the  OT&E  test 
team,  it  will  be  provided  by  mutual  agreement  of  the  Test  Director  and  tlie 
OT&E  Test  Director  (17:6). 

The  last  statement  hints  at  a most  important  point  that  \ias  con- 
sidered by  the  Vforking  Group  to  be  basic  in  any  test  program.  The  Test 
Director  and  OT&E  Test  Director  must  develop  a close  working  relationship 
with  each  other  right  frcm  the  very  start  if  the  program  is  to  stay  within 
cost  and  schedule  constraints.  Clearly  defined  responsibilities  and  early, 
coordinated  planning  are  vital. 

OT&E  Test  Program  Management 

The  question  of  who  manages  the  OT&E  programs  was  decided  by  the 
Working  Group  as  follows.  The  regulation  now  restricts  AFTEC  to  only 
major  programs  (as  defined  in  DoDD  5000.1,  (11:2))  and  certain  HQ  USAF- 
designated  non-major  programs;  all  other  non-major  programs  are  managed 
by  the  using  ccmmands.  However,  AFTEC  must  approve  test  plans  and  coitment 
on  final  reports  of  the  latter  non-major  programs.  In  this  way,  the 
Working  Grotp  further  limited  AFTEC' s involvement  in  OT&E,  yet  allowed 
for  their  independent  assessment  of  all  OT&E. 

Concerning  what  the  management  of  OT&E  includes,  the  regulation 
specifically  charges  AFTEC  (or  the  designated  f-lAJCfyi)  to  plan,  direct. 
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cx>nduct,  control  and  independently  evaluate  and  report  on  OT&E  by  OT&E 
personnel.  The  Working  Groi:^)  also  enphasized  the  responsibility  for 
independent  evaluation  as  opposed  to  independent  testing.  As  stated 
earlier,  separate  testing  cannot  be  conducted  on  many  programs  because  of 
time  and  money  constraints. 

As  a result,  the  Working  Group  changed  the  designation  from 
"combined  OT&EyiOT&E  test  program"  (16:6)  to  just  "combined  test  program" 
(17:6).  This  change  further  enphasizes  the  subtle  yet  important  concept 
that  on  a ccntoined  program  it  is  not  the  evaluation  that  is  combined,  but 
the  testing.  A separate  evaluation  is  the  real  task  of  the  independent 
test  agency. 

HQ  AFSC  has  encouraged  combined  developmental/operational  test 
programs  viienever  practicable  and  cost  effective,  as  opposed  to  a separate 
DT&E  followed  by  a separate  lOT&E  (19:23).  While  separate  lOT&E  is 
usually  best  for  small,  low  cost  articles,  it  is  often  prciiibitively  ex- 
pensive for  major  systems  like  aircraft. 

In  combined  programs,  the  lOT&E  requirements  must  be  accurately 
defined  and  blended  into  the  the  DT&E  effort.  Evaluation,  as  well  as 
testing,  requires  good  planning  due  to  the  high  cost  of  test  data: 

It  is  a mistake  to  believe  that  combined  test  events 
conpromise  the  independent  evaluation  of  a system.  But  the 
requirement  for  careful  planning  for  the  evaluation  of 
combined  test  data  is  obvious.  A common  problem  in  the  peist 
stemmed  from  poorly-defined  user  test  requirements  which 
led  to  a test  event  that  failed  to  provide  needed  user  data. 
Significant  inprovement  in  this  area  should  result  from 
AFTECfe  involvement  in  operational  test  planning  (19:24). 
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Innovative  OTaiE  Management  Concepts; 

Several  steps  were  taken  to  give  APIBC  increased  control  over  their 
assigned  responsibilities  and  at  the  same  time,  stronger  involvement  in 
the  T&E  process.  Early  participation  was  felt  necessary  to  provide  con- 
tinuity after  Required  Operational  Capability  (ROC)  validation  and  to 
adequately  plan  the  test  schedule.  As  a result,  the  Wbrking  Groi:^  provided 
for  OT&E  participation  in  the  conceptual  and  validation  phases  including 
test  program  planning  and  for  AFTEC  personnel  located  in  the  system  pro- 
gram offices  (SPOs)  and  contractor  facilities.  In  addition,  the  AFTEC 
OT&E  Test  Director  may  exercise  authority  to  rate  the  performance  of  key 
r^AJCOM  OT&E  test  team  personnel  (if  agreed  to  by  the  affected  MAJCQM) 

(15:2) . This  authority  was  reconrtended  by  the  Working  Group  to  give  AFTEC 
further  increased  control  over  the  CT&E  test  team  members. 

Responsibilities  of  the  Test  Director 

Because  the  implementing  command's  Test  Director  is  the  direct  re- 
presentative of  the  program  manager  on  the  test  team,  his  respxxisibilities 
were  made  to  coincide  very  closely  with  those  of  the  program  manager  r 

the  implementing  cotmand  provides  a test  director  vdo  is 

responsible  for  DT&E,  for  integrating  combined  test  events  and 
for  insuring  that  resources  are  made  available  to  carry  out  the 
combined  test  program  (17:6). 

Since  the  program  manager  has  overall  respcxisibility  for  a program 
(excep>t  OT&E  management) , it  follows  that  the  Test  Director  has  overall 
responsibility  for  the  test  portion  of  the  program  (except  OT&E  management) . 
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Relationship  Between  the  OT&E  Test  Director  and  the  Test  Director 

In  the  earlier  version  of  the  regulation,  the  OT&E  test  director  on 
a contoined  developnental/operational  test  team  had  been  loosely  referred 
to  as  the  "deputy  director"  for  lOT&E  (16:6).  This  title  had  become  some- 
v>tiat  of  an  issue,  and  the  Working  Group  decided  to  change  the  title  for 
several  reasons: 

1.  AFTBC  did  not  like  the  term  "deputy"  because  it  implied 
that  the  deputy  director  worked  for  the  test  director.  AFTEC 
viewed  this  as  a violation  of  their  basic  requirement  for 
independence . 

2.  Members  of  AFSC  did  not  like  "deputy"  either,  but  for 
different  reasons.  The  term  "deputy"  implied  that  the 
deputy  director  would  take  charge  of  the  entire  test  pro- 
gram if  the  test  director  were  TOY  or  became  ill.  Certainly 
the  title  was  never  intended  to  be  interpreted  in  this 
manner,  but  it  still  gave  people  as  uneasy  feeling. 

3.  The  title  was  incorrect  in  specifying  only  lOT&E 
because  the  deputy  director  usually  continued  his 
participation  into  the  FOT&E  phase  of  testing. 

As  a result,  the  Working  Group  decided  upon  the  title  of  "OT&E  Test  Di- 
rector" vAiich  is  appropriate  for  both  separate  and  combined  test  programs. 
The  new  title  provides  the  desired  degree  of  independence,  deletes  any 
possibility  of  assuitdng  the  Test  Director's  role,  ard  indicates  the  cor- 
rect generic  name  vAiich  includes  both  lOT&E  and  FOT&E.  Nevertheless,  the 
Test  Director  (from  the  developing  ccnmand)  has  overall  responsibility 
for  acccrplishing  the  test  progr^.  Besides  his  responsibilities  listed 
above,  he  has  responsibility  for  the  safe  conduct  of  all  tests  v^ch 
includes  "approval  authority  for  all  DT&E  test  events  and  approval  au- 
thority to  insure  that  OT&E  events  comply  with  safety  standards"  (17:7). 
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Misconceptions  of  Ccmbined  Test: 


Except  for  the  sanantic  change  concerning  the  OT&E  Test  Director,  the 
basic  relationship  between  the  oor±)ined  test  team  members  ranains  un- 
changed from  the  1975  to  the  1976  revisions  of  APR  80-14.  Yet,  concerns 
were  beginning  to  surface  even  before  the  regulation  was  published.  A 
letter  fron  the  Vice  Chief  of  AFSC  alleged  that  "dividing  the  test  effort 
into  two  distinct  teams  is  detrimental  to  truly  canbined  and  integrated 
testing"  (7).  Furthermore,  confusion  over,  and  misinterpretation  of  this 
section  (paragraph  17)  of  the  regulation  is  still  evident.  A recent  stu- 
dent report  fron  one  of  the  military  professional  schools  states,  "Para- 

i 

graph  17 seems  to  negate  any  sort  of  team  concept  in  canbined  test- 
ing"  ''Joint  planning  is  not  mentioned" "Unfortunately 80-14 

(does  not  specify)  an  individual  who  has  overall  test  program  responsi- 
bility" (24:14). 

In  refuting  these  misinterpretations,  it  should  be  mentioned  that  one 
of  the  basic  objectives  in  the  revision  was  to  shorten  the  regulation  by 
cleaning  up  the  verbosity  and  redundancy  of  the  earlier  revision.  Thus, 
the  Marking  Group  took  great  effort  in  stating  a particular  guidance  only 
once  in  the  regulation,  unless  it  was  absolutely  necessary  that  the  guid- 
ance be  repeated  elsevAiere  for  the  sake  of  clarity  or  emphasis.  Thus, 
because  certain  direction  is  not  found  in  one  paragraph  does  not  mean  that 
that  particular  direction  is  not  elsewhere  in  the  regulation. 

For  instance,  paragraph  7 states,  "When  the  implementing  Oonmand 


22 


begins  test  program  planning,  the  appropriate  MAJCOMS  and  agencies  will 
participate,  as  required  (17:4)."  This  joint  planning  is  applicable  to 
oont)ined  testing  discussed  later  in  the  regulation  and  is  certainly  inplied 
in  paragraph  17  which  states,  "the  planning  should  assure  that  ccxnbined 
testing  wDuld  fulfill  developmental  and  operational  test  objectives" 

(17:6) . Hew  else  could  planning  adequately  assvire  the  T&E  objectives 
without  having  "joint  planning?" 

Response  to  the  concerns  over  division  of  the  test  effort  into  two 
distinct  teams  and  JTF's  without  an  overall  director  have  already  been 
discussed  eairlier  in  this  text.  It  might  be  added  that  referring  to 
cor±)ined  test  teams  as  Joint  Test  Forces  (JTF)  does  not  follow  the  ter- 
minology used  in  AFR  80-14.  Joint  programs  are  defined  as  those  that  are 
conducted  with  other  Services,  U.S.  agencies  or  foreign  governments 
(17:3) , wliereas  v4iat  AFSC  new  calls  JTF  programs  are  defined  in  AFR  80-14 
as  "combined  test  programs"  (17:6).  Logically,  a more  appropriate  title 
for  the  test  team  would  be  Combined  Test  Force  (GIF) . Use  of  a more 
correct  title  in  denoting  the  type  of  program  is  becoming  increasingly 
inportant  as  the  number  of  joint  programs  increases  with  the  other 
Services  and  foreign  governments.  The  cruise  missile  and  F-16  programs 
are  examples  of  each. 

OT&E  in  Modification  Programs 

Ihaugh  AFSC  is  usually  the  implementing  conmand,  AFLC  is  also  involved 
in  a number  of  acquisition  and  modification  programs.  In  considering  OT&E 
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in  ntxlificaticn  programs,  AFLC  is  tasked  with  identifying  in  coordination 
with  APIEC  those  modifications  that  may  require  the  satisfaction  of  OT&E 
objectives  beyond  normal  modification  testing  or  proofing.  In  other  words, 
if  a modification  program  involves  sufficient  risk,  AFLC  should  recortmend 
to  HQ  USAF  that  OT&E  be  conducted  (17:10) . The  purpose  behind  this  addi- 
tion to  the  regulation  is  to  again  assure  that  adequate  testing  is  done 
before  a system  becomes  operational. 

Program  Reviews 

The  Chief  of  Staff's  (CSAF)  desire  to  give  AFTEC  more  clout  was  further 
satisfied  in  another  way:  The  Ccnmander  of  AFTEC  may  request  AFTEC  attend 
specific  program  reviews  besides  providing  OT&E  information  to  the  CSAF  in 
preparation  for  decision  Milestone  reviews  (15:2). 

Operating  and  Sup^rt  (O&S)  Costs 

To  clarify  managotient  of  O&S  costing,  the  Working  Group  expanded  the 
regulation's  direction  in  vhat  had  formerly  been  known  as  cost  of  owner- 
ship. The  major  cortmands  are  to  develop  and  provide  the  O&TE  test  agency 
the  models,  techniques,  or  cost  element  data  used  to  construct  estimates 
of  O&S  costs  (17:9).  These  are  to  be  used  during  OT&E  to  acquire  infor- 
mation which  can  improve  and  validate  O&S  cost  estimates  and  identify 
system  parameters  or  deficiencies  vhich  impact  O&S  costs. 

Foreign  Military  Sales  (FMS) 

With  the  increased  proportion  of  EWS  cases  involving  narer  weapons 
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systene,  FMS  considerations  and  requirements  are  becaning  more  iirportant 
in  the  earlier  phases  of  a system's  life  cycle.  At  first  the  Working 
Grovp  thought  to  require  all  FMS  testing  to  comply  with  AFR  80-14.  How- 
ever, the  requesting  foreign  countries  restricted  by  political,  schedul- 
ing, or  even  monetary  constraints  might  have  good  reason  not  to  fund 
testing  under  our  rules;  therefore,  the  regulation  was  made  less  re- 
strictive. T&E  for  FT1S  programs  will  conp]y,  where  feasible,  with  the 
principles  of  AFR  80-14  (17:3). 

Sunmary  of  Changes 

A concise  sunmary  of  the  major  changes  to  AFR  80-14  and  23-36  is 
presented  in  the  following  matrix.  The  regulations  are  added  as  i^pendices 
A and  B to  aid  in  review  of  the  subject. 
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NEEDED  REVISIONS  PREVIOUS  REES.  CURRENT  REES.  DoD  DIRECTIVES 


SECTION  IV 


EFFECTS  OF  NEW  DoD  DIRECTION  PERTAINING  TO  USAF  T&E  POLICY 

New  Guidance 

In  1976,  the  Office  of  Managanent  and  Budget  (CMB)  established 
policies  to  be  folla^^d  by  all  Federal  Government  executive  branch  agen- 
cies in  the  acquisition  of  itajor  systons.  This  direction  has  beccxte 
kncwn  as  CMB  Circular  No.  A-109,  (8)  and  has  had  the  first  major  inpact 
on  acquisition  policy  since  the  Blue  Ribbon  Defense  Panel  came  vp  with 
113  substantive  recaimendations  in  1970  (22:5).  Subseq^aently , DoD  began 
reviews  of  DoEOs  5000.1,  5000.2,  and  5000.3  to  implement  the  new  guidance. 

Revision  of  DoD  Directives 

DoDD  5000.3,  Test  and  Evaluation,  was  to  have  been  published  soon 
after  DoDDs  5000.1  and  5000.2  were  issued  in  January  1977,  but  the  length 
of  the  coordination  process  followed  by  plans  to  reorganize  the  Director 
of  Defense  Research  and  Engineering  (DDR&E)  have  delayed  ccarpletion  of 
this  revision  (29).  The  following  surmiary  will  focus  primarily  on  the 
changes  of  the  T&E  portions  of  these  directives  and  how  they  relate  to 
Air  Force  T&E  policy. 

The  main  purpose  behind  the  revision  of  DoEOs  5000.1  and  5000.2  was 
bo  introduce  the  Mission  EQsement  Need  Statement  (MENS)  and  Milestone  0 
(Program  Initiation) . Briefly,  the  MENS  is  a statement  prepared  by  a 
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DoD  Conponent  to  identify  and  sipport  the  need  for  a new  or  iitproved 
mission  capability  as  opposed  to  a requiranent  for  specific  hardware. 
Approval  of  the  MENS  by  the  Secretary  of  Defense  with  subsequent  direc- 
tion to  one  or  mure  CcmDonents  to  exolore  and  develop  system  concepts  to 
satisfy  the  approved  need  is  called  Milestrme  O (12:6).  DoDD  5000.3 
ocrplements  the  other  two  directives  and  to  be  in  agreenent  with  them. 

Was  put  in  revision  slightly  behind  the  two.  As  mentioned  earlier,  how- 
ever, the  draft  DoDD  5000.3  is  still  under  study.  Though  there  are  a 
nixnber  of  changes  to  the  directives,  most  of  them  either  relate  to  the 
basic  purpose  for  revising  the  docunents  or  sinply  clarify  previous 
policy.  As  a result,  there  are  no  major  changes  in  the  area  of  test  and 

I 

evaluation  in  DoDD  5000.1  or  5000.2.  However  the  draft  DoIX)  5000.3  has 
scare  important  revisions,  and  there  are  several  vi^ich  need  further  study 
and  resolvement  before  the  draft  becomes  final. 

DT&E  No  Longer  to  Assess  Military  Utility? 

In  starting,  an  iirportant  deletion  was  noticed  in  the  basic  de- 
finition of  OT&E  and  DTSE  in  the  new  direch:ives.  Though  DoDD  5000.1 
defines  OT&E  (ll:Encl  1:2)  as  being  conducted  to  estimate  a system's 
military  utility,  operational  effectiveness  and  operational  suitability 
(referenced  to  the  1973  DoDD  5000.3),  the  draft  DoED  5000.3  drops  the 
term  "military  utility"  from  discussions  and  definitions  of  both  DT&E 
and  OT&E  which  were  in  the  older  version  (13:2,3).  For  instanoe,  the  draft 
DoDD  5000.3  has  added  a Glossary  of  definitions  in  v;hich  DT&E  is  defined 
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as: 


that  test  and  evaluation  conducted  to  assist  the 

engineering  design  and  develcpnent  process  and  verify 
attainraent  of  technical  performance  specifications  and 
(±>jectives. 

floreover,  DoDD  5000.1  fails  to  give  any  definition  at  all  for  DT&E  in  its 
definition  section. 

What  does  all  this  lead  to?  It  appears  that  DT&E  is  not  to  be  in- 
volved in  estimating  military  utility  if  the  draft  DoDD  5000.3  is  ap- 
pro'/ed.  This  is  unfortunate  because,  as  mentioned  in  Section  II,  feed- 
back concerning  military  utility  by  the  developmental  test  pilot  to  the 
OT&E  test  team  could  lead  to  irtproved  OT&E  missions  and  possibly  reduce 
OT&E  reqtiired  data. 

Furthermore,  an  additional  viewpoint  provided  to  the  decision  makers 
could  reduce  the  risk  of  parochialism  or  biased  reporting  that  can  easily 
occur  in  the  present  environment  where  there  are  just  a few  pilots  on  a 
test  program.  For  instance,  under  the  DoD  requirement  to  choose  an  average 
pilot  for  cperational  tests,  the  background  of  the  individual  selected  may 
be  limited  to  2 or  3 aircraft  as  opposed  to  that  of  the  developmental  test 
pilot  with  a minimum  of  20-30  aircraft.  If  the  cperational  pilot  is  con- 
ditioned to  poor  characteristics  in  his  past  aircraft  (such  as  weak  brakes, 
inadequate  visibility  or  a gvmsight  that  jitters)  and  the  test  article 
ejdiibits  much  the  same  characteristics,  he  is  liable  to  judge  them  as 
"normal"  or  "good."  The  developmental  test  pilot,  on  the  other  hand,  is 
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able  to  give  a itore  valid  assessinent  of  its  true  characteristics  by 
ccnparison  to  a broad  spectrum  and  number  of  aircraft. 

AFIEC  to  Participate  in  Planning  of  DT&E? 

A problan  area  that  needs  to  be  re-worded  is  in  the  draft  as  follcuvs: 

The  Corponent  OT&E  Agency  will  parti ci^te  in  planning  of 
DT&E  (underlining  added)  to  ascertain  v^ich  ^rtion  of  DT&E 
will  contribute  to  the  acccirplishment  of  OT&E  objectives 
(14:6). 

The  thought  is  there  but  the  interpretation  that  results  is  not  correct. 

AETEC  does  not  share  in  planning  OT&E.  The  main  idea  behind  the  referenced 

sentence  is  that  coordinated  test  planning  is  necessary  in  a program.  To 

prevent  misinterpretation,  a better  choice  of  words  would  be: 

The  Ccnponent  OT&E  Agency  will  participate  with  the 
Conponent's  development  agency  during  DT&E  planning  to 
ascertain  which  portion  of  OT&E  viill  contribute  to  the 
accatplishment  of  OT&E  objectives. 

Seme  Changes  Led  by  the  Air  Force 

It  is  noteworthy  that  there  are  a covple  areas  in  which  Air  Force 
policy  has  led  DoD  in  its  revision  of  DoDD  5000.3.  The  most  important 
is  early  involvement  by  OT&E  personnel.  The  draft  directive  is  following 
suit  and  specifies  that  OT&E  will  comnenoe  as  early  as  possible  in  the 
development  cycle.  This  early  involvement  is  part  of  the  overall  effort 
to  load  the  front  end  of  the  acquisition  process  with  greater  planning 
and  visibility  in  order  to  decrease  production/O&S  costs  later  on. 

Furthemore,  the  draft  defines  lOT&E  as  the  Air  Force  does: 
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OT&E  conducted  prior  to  the  Milestone  III  decision  is 
designated  Initial  Operational  Test  and  Evaluation  (lOT&E). 

m 

m. 

It  has  no  effect  on  the  Air  Force,  but  the  Anry  cind  Navy  should  change 
their  OT/DT  and  OPEVAL/TECHEVAL  terminology. 

Finally,  joint  programs  with  other  Conponents  are  given  increased 
atphasis  in  the  draft  including  the  introduction  of  Joint  Operational  Test 
and  Evaluation  (JOT&E) . This  emphasis  gives  further  reason  to  call  a USAF 
ocmbined  developmental/operational  test  team  a COrbined  Test  Force  rather 
than  a Joint  Test  Force,  as  proposed  earlier  in  Section  III. 

Combined  Versus  Independent  Testing 

Another  inpDrtant  change  evident  in  the  draft  DoDD  5000.  3 is  the 

softening  of  current  DoD  philosophy  that  states  operational  testing 

should  be  separate  from  development  testing  (13:4).  Admiral  Kollmorgen, 

Asst  Director  of  the  DDR&E  office  responsible  for  DoEO  5000.3,  indi.cates 

that  the  new  T&E  policy  changes  oiphasizing  earlier  cperationail  testing, 

valid  reliability  goals  and  combined  developmental  and  operational  testing 

are  expected  to  improve  the  efficiency  of  the  acquisition  process  (23:3). 

Ihe  draft  version  reflects  this  new  policy  by  stating: 

Development  testing  and  operational  testing  may  be  contoined 
v^iere  clearly  identified  and  significant  cost/time  benefits 
would  result... (14:7). 

This  ^proach  is  much  more  realistic  and  clearly  a necessity  in  most 
major  programs. 

However,  the  same  paragraph  in  the  revision  takes  a firmer  view 


33 


tcward  independent  testing  by  the  CT&E  agency. 

As  a normal  practice  the  operational  tests  supporting  a 
production  decision  will  be  conducted  independently  by 
the  OT&E  organization  (14:8). 

With  eitphasis  on  canbined  test  programs  but  independent  testing,  there 
will  have  to  be  close  coordination  and  planning  between  the  Test  Director 
and  the  OT&E  Test  Director.  To  reduce  the  possibility  of  schedule  stretch- 
out due  to  independent  testing,  the  Test  Directors  vdll  need  to  maximize 
the  test  missions  \diere  data  is  ccittnon  ti  both,  yet  ensure  that  the  re- 
quired independent  testing  car  be  accormodated  separately. 

Long-Lead  Items 

Ihe  ability  to  order  long- lead  itans  has  been  a problem  with  Program 
Managers  ever  since  concurrent  development  got  a bad  nane.  DoDD  5000.1 
ncxv  gives  the  Secretary  of  Defense  authority  to  approve  long- lead  pro- 
duction items  at  liilestone  II.  In  addition,  the  draft  DoIX)  5000.3  allows 
OT&E  to  provide  inputs  to  decisions  made  in  Milestone  II  for  long- lead 
items  or  for  limited  production. 

1Et.JP 

The  Test  and  Evaluation  Master  Plan  (TEMP)  section  in  DoED  5000.3 
has  been  ejqjanded  in  the  draft.  Besides  indicating  that  the  TEMP  will 
prescribe  carplianoe  with  environmental  activities  and  will  include 
electromagnetic  ocnpatibility  considerations,  the  TEMP  will  be  submitted 
to  OSD  "for  coordination  and  reviev/s. . .relative  to  the  major  milestones." 


i 
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This  requirement  appears  to  be  removing  some  of  the  managanent  authority 
from  the  program  manager,  and  since  DoDD  5000. 1 calls  for  the  decentraliza- 
tion of  responsibility  for  program  managarent,  there  seems  to  be  a con- 
flict between  the  two  directives.  If  the  draft  DoDD  5000. 3 is  approved, 
the  program  iranager  can  look  forward  to  the  possibility  of  what  has  been 
termed  "micrananagement"  frcm  OSD.  In  addition,  this  new  review  could 
further  lengthai  the  front  end  of  acquisition  programs  vAiere  a growth 
has  been  noticed  over  the  past  few  years. 

Softtvare  Enphasized 

There  is  a whole  new  section  in  the  draft  DoE©  5000.3  concerning 
T&E  of  conputer  software  plus  references  to  software  elsewhere  in  the 
draft.  At  first  glance  it  appears  questionable  whether  software  is  so 
unique  frcm  other  special  programs,  siKh  as  satellites,  siimilators  or 
ground  power  carts,  that  it  requires  separate  attention.  Hoirever,  this 
special  emphasis  is  necessary  because  major  programs  have  floundered  as 
a result  of  inadequate  development  of  software  (28).  Software  has  be- 
came extremely  expensive  in  itself  aiid  therefore,  must  be  given  considered 
attention  alcxig  v/ith  other  system  carponents  in  the  development  process. 

Flexible  Management 

Finally,  a review  of  DoDD  5000. I's  Policy  section  indicates  that 
successful  management  of  systen.  acquisition  depends  upon  recognition 
that  programs  are  different  and  require  managenent  flexibility  (11:12). 

This  ackrxwledqement  reinforces  the  Air  Force's  philosophy  used  in 
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writing  80-14  that  there  are  no  step-by-step  methods  in  program 
management. 
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SExrrioN  V 


SUM>1ARY  AND  REXXMMEMDATIONS 

In  the  foregoing  Sections,  insight  was  given  as  to  the  inportant 
motivators  behind  current  USAF  test  and  evaluation  policy,  the  rationale 
used  in  establishing  this  policy,  and  recent  expected  changes  to  it.  This 
study  should  prove  valuable  to  the  program  manager  (PM)  in  giving  a clearer 
understanding  of  the  T&E  regulations  and  ultimately  assist  in  program 
management. 

Besides  the  additional  guidance  presented  in  the  earlier  Sections, 
there  are  also  a number  of  recarmendations  and  conclusions  that  can  be 
derived  frcm  these  discussions. 

1.  Be  Ccureful  of  biased  interpretation. 

Though  this  reconmendation  applies  to  all  regiHations  in  general, 
it  is  especially  important  in  AFRs  80-14  and  23-36  where  APTEC  has  been 
tasked  to  assume  an  adversary  role.  In  addition,  test  teams  are  one  of 
the  most  oonplex  organizations  as  far  as  management  is  concerned  because 
of  the  diversity  in  backgrounds  and  paurent  ccmands.  There  is  a tendency 
to  interpret  regulations  in  terms  of  one' s prejudices  and  biases  and  to 
fit  the  interpretation  to  benefit  one's  paurticular  aspect  of  a management 
problem  while  disregarding  an  equally  valid,  but  opposite,  interpretation. 
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In  addition  to  the  PM's  own  biases,  the  nust  be  aware  of  the  bias 
possibility  in  other  program  offices  and  in  test  team  mentoers,  and  weigh 
opposing  interpretations,  looking  first  as  to  which  is  the  more  correct, 
and  second,  hew  the  program  will  be  affected. 

2.  Be  flexible. 

AFRs  80-14  and  23-36  were  written  to  provide  general  rather  than 

specific  guidance  for  a purpose  - to  allow  the  greatest  flexibility  in 

program  management.  DoCO  5000.1  states: 

Successful  management  of  system  acquisition  dep>ends  upon 
caipetent  people,  defined  responsibilities  and  authority, 
realistic  objectives,  rational  priorities  and  recognition 
that  programs  are  different  and  require  management  flexi- 
bility (11:2) . (Underlining  added  for  enph^is') 

The  PM  should  take  advantage  of  the  "generalities"  discussed  earlier  in 
T&E  policy  to  tailor  the  program  to  its  unique  requirements.  These 
genercilities  can  provide  much  of  the  flexibility  required  for  a success- 
ful program. 

3.  There  are  no  cook-book  methods  in  program  management. 

Since  it  is  recognized  in  recenmendation  2 that  programs  are 

different  and  unique,  then  it  follows  that  here  are  no  step>-by-step  pro- 
cedures in  managing  a program.  This  conclusion  results  from  the  coitinon 
belief  that  acquisition  management  does  not  lend  itself  to  scientific 
regimentation,  or  as  Henry  Boettinger  put  it,  "In  sheer  banality,  few 
statements  exceed  the  assertion  that  management  is  an  art"  (25:54). 

Again,  the  "gaieralities"  in  AFRs  80-14  and  23-36  piermit  the  PM  maxinum 
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latitude  within  DoD  and  USAF  policy  constraints  in  which  to  practice  his 
craft. 

4.  Recognize  vdiat  is  not  said  in  regulations. 

This,  too,  applies  to  all  regulations  and  guidance,  not  just 
AFRs  80-14  and  23-36.  Too  often  there  is  a tendency  to  criticize  the  policy 
that  is  found  in  regulations  vdiile  ccatpletely  neglecting  \^tet  is  not  said. 
Where  there  is  no  policy  given,  the  PM  can  gain  additional  latitude  and 
flexibility  in  conducting  his  program  as  necessary  without  restrictions. 

The  smart  manager  will  take  advantage  of  these  lansaid  possibilities  and 
tailor  his  program  accordingly. 

5.  Early  planning  is  vital. 

Fincilly,  but  foremast,  early  planning  is  necessary.  The  Air 
Force  stressed  this  need  early  in  USAF  policy,  and  now  DoD  is  also 
eitphasizing  its  inportance  (23:7).  Because  of  the  numero\:is  MAJCCMs  in- 
volved during  an  aoquisiticxi , close  coordination  between  all  agencies 
is  required  to  keep  a program  within  cost  and  on  schedule.  Early  plan- 
ning between  DT&E  and  OTiiE  p)ersonnel  is  especially  important  in  ootnbined 
OT&E/IOT&E  test  programs.  The  conpetition  is  severe  for  resources  in 
order  to  accomplish  desired  ctojectives  under  schedule  constraints. 

In  addition,  the  personalities  of  the  Test  Direcrtor  and  the  OT&E 
Test  Director  are  very  inportant.  These  individuals  must  be  able  to  wrk 
closely  together  without  friction  during  planning,  control,  and  conduct 
of  the  test  program  and  also  in  resolving  differences  as  they  arise.  The 
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French  author  and  politician  Jean-Jacques  Servan-Schreiber  sums  up  the  idea: 
"Management  is  the  art  of  arts  because  it  is  the  organizer  of  talent"(25:55). 
The  Test  Director  and  the  OT&E  Test  Director  must  strive  to  be  true  artists 
in  working  not  only  together  but  also  with  their  test  teams  because  they 
both  have  the  common  and  ultimate  goal  of  providing  a better  weapon  system. 
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SECTION  A— SCOPE  AND  CONCEPTS 

1.  Scope.  Air  Force  test  and  evaluation  (T&E)  is 
conducted  in  four  areas  of  effort:  T&E  in  the 
technology  base  program;  T&E  in  the  acquisition 
process;  T&E  activities  within  Air  Force  en- 
gineering service  resources;  and  joint  programs 
with  other  governments  or  with  other  US  Gov- 
ernment agencies.  The  emphasis  of  this  regula- 
tion is  on  T&E  during  the  acquisition  process  (see 
chart  in  attachment  2). 

2.  Concepts: 

a.  Development  and  Acquisition  Programs. 
There  are  two  basic  kinds  of  test  and  evaluation: 
Development  Test  and  Evaluation  (DT&E)  and 
Operational  Test  and  Evaluation  (OT&E).  DT&E 
and  OT&E  are  essential  parts  of  the  system  ac- 
quisition process,  and  may  occur  throughout  all 
phases  of  the  system’s  life  cycle.  The  primary 
purpose  of  all  T&E  in  the  acquisition  process  is  to 
estimate  the  military  utility  of  a system,  subsys- 
tem, or  item  of  equipment  (all  hereafter  referred 
to  as  a system). 

(1)  DT&E  is  conducted  to  demonstrate  that 
engineering  design  and  development  are  com- 
plete, that  design  risks  have  been  minimized,  and 
that  the  system  will  meet  engineering  and  opera- 
tional specifications.  DT&E  is  essentially  a de- 
tailed engineering  analysis  of  a system’s  per- 
formance (beginning  with  individual  subsystems 
and  progressing  through  a complete  system), 
where  system  design  is  tested  and  evaluated 
against  engineering  and  performance  criteria  by 
the  implementing  command. 

(2)  OT&E  is  conducted  to  estimate  a prospec- 
tive system’s  operational  effectiveness  and  opera- 
tional suitability,  and  to  identify  any  operational 
deficiencies  and  need  for  any  modifications.  In  ad- 
dition, OT&E  provides  information  on  organiza- 
tion, personnel  requirements,  doctrine,  and  tac- 
tics. OT&E  is  essentially  an  operational  assess- 
ment of  a system’s  performance  where  the  com- 
plete svstem  is  tested  and  evaluated  against  oper- 
ational criteria  (requirement  and  employment 
concepts)  by  personnel  with  the  same  qualifica- 
tions as  those  who  will  operate,  maintain,  and 
support  the  system  when  deployed. 

(3)  DT&E  and  OT&E  will  begin  as  early  as 
possible  in  a system’s  life  cycle  and  continue 
throughout,  as  necessary,  to  assess  the  system’s 
military  utility  and  educe  acquisition  risk. 


(a)  During  the  conceptual  and  validation 
phases  of  a system’s  life  cycle,  T&E  is  conducted 
to  demonstrate  feasibility,  to  minimize  design 
risks,  and  to  determine  design  alternatives  and 
trade-offs  required  to  best  achieve  program  ob- 
jectives. Tests  of  subsystems,  components,  or 
system  prototypes  are  conducted  to  develop  data 
on  which  to  base  a full-scale  engineering  de- 
velopment decision.  While  the  testing  emphasis 
during  this  period  is  on  the  DT&E,  a cadre  of  per- 
sonnel responsible  for  OT&E  will  participate  in 
the  conceptual  and  validation  phases,  and  will  re- 
port on  any  OT&E  conducted. 

(b)  During  the  full-scale  engineering  de- 
velopment phase,  T&E  progresses  from  subsys- 
tem and  component  checks  to  full-system  tests. 
The  objective  is  to  further  determine  that  design 
risks  are  minimized,  system  design  is  complete, 
and  if  the  military  utility  of  the  system  would  jus- 
tify production. 

1.  While  DT&E  will  require  heavy  em- 
phasis in  full-scale  engineering  development, 
OT&E  will  play  an  essential  role  in  the  first  major 
production  decision  for  a system.  A prerequisite 
for  this  decision  is  that  an  initial  phase  of  OT&E 
(called  lOT&E)  be  accomplished  to  provide  an  es- 
timate of  expected  system  operational  effective- 
ness and  suitability. 

2.  lOT&E  ends  with  the  first  major  pro- 
duction decision,  and  thereafter,  all  operational 
testing  is  called  follow-on  operational  test  and 
evaluation  (FOT&E).  These  T&E  relationships 
are  shown  in  attachment  2. 

3.  Operational  testing  should  be  separate 
from  development  testing.  However,  ea’  y phases 
of  OT&E  may  need  to  be  combined  with  develop- 
ment testing  where  separation  would  cause  delays 
involving  unacceptable  military  risk,  or  would 
cause  an  unacceptable  increase  in  the  acquisition 
cost  of  the  system. 

(c)  During  the  production  and  deployment 
phases,  test  and  evaluation  is  conducted  to  refine 
system  operating,  maintenance,  and  support  con- 
cepts; develop  new  operating  concepts;  deter- 
mine the  need  for  and  assess  the  technical  and  op- 
erational effectiveness  of  system  modifications; 
insure  the  effectiveness  of  the  manufacturing 
process,  equipment  and  procedures  (Production 
Qualification  and  Acceptance  Tests);  and  deter- 
mine if  the  system  is  and  remains  in  a relatively 
stable  configuration  for  operational  use.  While 
DT&E  may  continue  at  a relatively  high  level  in 
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the  production  and  deployment  phases,  OT&E 
activities  will  normally  increase  in  scope. 

(d)  In  addition  to  OT&E  application  as  de- 
scribed in  (a),  (b)  and  (c)  above,  there  are  pro- 
grams for  which  OT&E  must  be  performed  even 
though  there  has  not  been  a foregoing  lOT&E. 
This  type  of  testing  is  referred  to  as  “qualification 
OT&E.”  Examples  of  this  are  those  Class  IV  and 
V modifications,  simulators,  commercially  de- 
veloped items,  and  other  items  which  may  require 
no  development,  per  se,  and  therefore,  no  DT&E 
or  lOT&E,  and  which  may  or  may  not  be  as- 
sociated with  a production  decision. 

b.  Technology  Base  Programs.  Test  and  evalua- 
tion is  also  conducted  in  the  research,  exploratory 
development,  and  advanced  development  subdivi- 
sions of  the  Air  Force  Technology  Base  Program. 
In  these  phases  of  technology  development,  test 
and  evaluation  is  conducted  to  verify  hypotheses; 
measure  phenomena;  develop  new  techniques, 
procedures,  and  materials;  and  estimate  the  mili- 
tary utility  of  new  components,  subsystems,  or 
technologies. 

c.  Engineering  Services.  Engineering  Services 
(including  testing)  are  unique  capabilities  pro- 
vided primarily  by  AFSC  T&E  activities  which 
are  made  available  to  support  the  T&E  efforts  of 
government  agencies  (through  the  AFSC  En- 
gineering Services  Program)  and  government- 
sponsored  agencies  (AFR  80-19,  Support  of  Non- 
governmental Test  and  Evaluation).  These  en- 
gineering services  are  not  necessarily  a part  of 
any  Air  Force  development  or  acquisition  pro- 
gram, but  are  provided  under  specific  support  ar- 
rangements. They  include  technical  support, 
facilities  and  knowledge  not  available  elsewhere. 

d.  Joint  Programs: 

(1)  When  the  Air  Force  is  lead  service  in  a 
joint-service  acquisition  program,  T&E  is  con- 
ducted as  outlined  in  this  regulation. 

(2)  When  another  service  or  agency  is  the 
lead  service,  T&E  is  conducted  in  accordance 
with  an  agreement  between  the  Air  Force  and 
other  involved  services  or  agencies. 

(3)  When  joint  service  testing  is  sponsored 
by  ODDR&E,  T&E  is  conducted  in  accordance 
with  agreements  between  ODDR&E,  the  Air 
Force,  and  other  involved  services. 

(4)  Joint  testing  with  other  governments  will 
be  in  accordance  with  country-to-country  agree- 
ments. 

(5)  Test  and  evaluation  associated  with 
Foreign  Military  Sales  programs  will  also  comply, 
where  feasible,  with  the  principles  of  this  regula- 
tion. 

SECTION  B— POLICY  GUIDANCE  ON  T&E 
MANAGEMENT 


3.  Relationship  Between  the  System  Program 
Manager  and  AFTEC.  The  program  manager 
has  overall  responsibility  for  a system  acquisition 
program  (except  the  management  of  OT&E).  His 
responsibility  includes  managing  DT&E,  incor- 
porating the  OT&E  requirements  into  the  test 
program,  and  providing  support  for  OT&E  as  ap- 
propriate. AFTEC  (or  the  designated  major 
command  (MAJCOM)  ) has  responsibility  for 
managing  the  OT&E  in  a major  acquisition  pro- 
gram. That  is,  AFTEC  (or  the  designated  MAJ- 
C;OM)  will  plan,  direct,  conduct,  control,  and  in- 
dependently evaluate  and  report  on  OT&E. 

4.  Consolidating  Test  Events  and  Resources.  To 
minimize  duplication  of  testing,  test  events 
should  be  consolidated  to  the  extent  that  the  con- 
solidation does  not  alter  or  compromise  the  test 
purpose  and  objective.  Test  data  from  all  avail- 
able sources  will  be  used  to  the  extent  possible. 

a.  Tests  will  not  be  conducted  without  evalua- 
tion. However,  if  relevant  data  is  available  from 
earlier  tests  or  other  sources,  evaluations  may  be 
made  without  additional  testing. 

b.  Existing  Government-owned  T&E  capabili- 
ties (for  example,  ranges,  instrumentation,  and 
other  related  resources)  will  be  used  whenever 
possible.  However,  T&E  will  be  conducted  in  the 
most  realistic  operational  environment  that  is 
economically  prudent  and  feasible. 

5.  Documentation.  Program  documents,  such  as 
the  Decision  Coordinating  Paper  (DCP),  Program 
Memorandum  (PM),  Program  Management  Di- 
rective (PMD),  Test  Directive  (TD),  or  Program 
Management  Plan  (PMP),  must  include  a state- 
ment of  the  critical  questions  and  areas  of  risk  to 
be  resolved  by  T&E.  This  documentation  will  also 
provide  a summary  of  test  objectives,  schedules, 
and  milestones. 

a.  Each  DCP  should  discuss,  in  its  T&E  sec- 
tion, whether  or  not  the  .system  is  amenable  to 
multi-service  testing  or  two-sided  testing.  If  such 
testing  is  appropriate,  the  DCP  and  Test  and 
Evaluation  Master  Plan  (TEMP)  should  state  the 
intent  and  plans  for  its  conduct.  The  following 
definitions  apply: 

(1)  Multi-service  testing.  Testing  that  entails 
active  participation  (for  example,  personnel  and 
money)  of  more  than  one  service  during  the  de- 
velopment of  a weapon  system  potentially  appli- 
cable to  the  needs  of  more  than  one  Service. 

(2)  Two-sided  testing.  Testing  in  which  two 
systems  perform  in  adversary  roles.  This  could 
involve  either  multip'  j services  or  a single  serv- 
ice. 

b.  The  OT&E  and  DT&E  criteria  used  to  judge 
the  system’s  performance  must  be  based  on  the 
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military  requirement  and  on  the  operational 
employment  and  maintenance  concepts  prepared 
by  MAJCOMS  or  other  agencies  and  approved  by 
HQ  USAF.  These  criteria  must  be  reflected  in 
the  Test  and  Evaluation  Objectives  Annex 
(TEOA)  to  the  PMD;  see  attachment  3. 

6.  What  Systems  Are  To  Be  Tested: 

a.  New  systems,  including  conventional  arma- 
ments, nuclear  weapons  systems  (except  subsys- 
tems that  are  governed  by  joint  DOD-Energy 
Research  and  Development  Administration 
(ERDA)  agreement),  associated  equipment,  and 
commercial  items  will  be  tested  and  evaluated  in 
accordance  with  this  regulation. 

b.  All  modifications  which  fit  the  criteria  for 
major  programs  (as  defined  in  DODD  5000.1)  or 
modification  programs  that  require  RDT&E- 
funded  development  will  be  tested  and  evaluated 
in  accordance  with  this  regulation. 

(1)  If  no  RDT&E  funding  is  required,  or  if  the 
modification  does  not  fit  the  criteria  for  a major 
program,  the  engineering,  prototyping,  and  test 
and  proofing  will  satisfy  preproduction  T&E  re- 
quirements and  will  not  require  reporting  outside 
modification  channels,  unless  the  PMD  directs 
otherwise.  However,  in  these  programs,  the  im- 
pact of  the  modification  on  operational  effective- 
ness and  suitability  must  be  assessed  early  in  pro- 
gram planning. 

(2)  If  the  production  decision  requires  the 
satisfaction  of  OT&E  objectives  beyond  normal 
modification  testing  or  proofing,  HQ  USAF  may 
require  compliance  with  the  T&E  principles  out- 
lined here  and  require  reporting  as  specified  in 
paragraph  26. 

c.  For  one-of-a-kind  systems,  or  systems  that 
involve  the  development  or  procurement  of  only  a 
very  few  units  over  an  extended  period  (for 
example,  simulators),  the  T&E  procedures  out- 
lined here  will  apply.  Compatibility  with  existing 
or  planned  equipment  will  be  tested.  OT&E  will 
be  conducted  as  early  as  possible,  but  no  later 
than  final  acceptance. 

7.  Participation  in  Program  Planning.  When 
the  implementing  command  (or  MAJCOM  or  AF- 
TEC)  begins  test  program  planning,  the  appro- 
priate MAJCOMS  and  agencies  will  participate, 
as  required. 

a.  When  more  than  one  MAJCOM  or  agency  is 
involved  in  a given  OT&E  activity  or  program, 
the  program  directive  will  designate  the  lead 
command,  unless  separate  tests  are  necessary 
because  of  different  mission  requirements. 

b.  If  the  T&E  being  done  by  one  command  in- 
volves flying  an  aircraft  of  another  command,  or 


using  the  test  articles  of  another  command,  an 
agreement  between  the  commands  (or  the  coor- 
dinated test  plan)  will  assign  supervisory  respon- 
sibility under  the  various  test  arrangements. 
Specifically: 

(1)  The  appropriate  test  director  will  make 
sure  that  the  test  items  are  operated  within  cur- 
rent operational  limitations,  and  in  accordance 
with  all  of  the  pertinent  Air  Force,  major  com- 
mand and  test  force  directives  and  procedures. 

(2)  Accountability  for  aircraft  or  drone,  mis- 
sile, explosive,  ground,  and  nuclear  accidents  will 
be  determined  as  outlined  in  AFR  127^. 

8.  Priority  for  Allocating  Test  Articles  and 
Equipment.  The  allocation  and  delivery  of  test 
articles,  peculiar  test  equipment,  or  spares  for 
the  system  undergoing  test,  will  take  precedence 
over  any  production,  training  or  operational  re- 
quirements for  system  equipment  and  personnel, 
unless  directed  otherwise  by  HQ  USAF.  Also, 
unless  directed  otherwise  by  HQ  USAF,  common 
items  will  be  supplied  according  to  the  DOD 
Priority  System. 

9.  Documenting  Waivers.  When  a T&E  program 
cannot  be  conducted  as  prescribed  here,  the  na- 
ture and  rationale  for  the  proposed  deviation 
must  be  specified  in  the  program  document  (for 
example,  the  Test  and  Evaluation  Master  Plan 
(TEMP)  ) which  must  be  approved  by  HQ  USAF. 
If  approved,  the  waiver  will  be  documented  in  the 
proper  program  directives. 

a.  If  the  waiver  concerns  T&E  outlined  in  a 
program  memorandum  for  a non-major  program, 
the  waiver  must  have  the  approval  of  the  Secre- 
tary of  the  Air  Force,  or  a designated  Assistant 
Secretary. 

b.  If  the  waiver  concerns  T&E  outlined  in  a 
DCP  for  a major  program,  the  waiver  must  have 
the  approval  of  the  Secretary  of  Defense. 

10.  International  Constraints.  Under  the 
Strategic  Arms  Limitation  (SAL)  Agreements, 
there  are  certain  constraints  which  restrict  the 
conduct  of  USAP'  development,  test,  and  opera- 
tional activities.  All  USAF  test  activities  will 
comply  with  these  restrictions  and  requirements, 
as  outlined  in  AFR  28-1. 

SECTION  C— CONCEPTS  AND  OBJECTIVES 
OF  T&E  PROGRAM  MANAGEMENT 

11.  What  Is  Involved  in  the  Development  Test- 
ing and  Evaluation  Phase: 

a.  To  determine  how  well  the  technical  and  op- 
erational requirements  which  were  specified  ir 
the  program  documentation  have  been  met. 
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DT&E  is  usually  performed  on  a model,  pro- 
totype, or  preproduction  article  (this  article  may 
also  be  used  for  lOT&E).  Therefore,  it  is  essen- 
tial to  plan  for  an  adequate  number  of  test  articles 
for  DT&E  and  lOT&E  which  are  fabricated  and 
configured  in  a manner  which  represents  the  an- 
ticipated production  item. 

b.  Sufficient  DT&E  must  be  accomplished  be- 
fore every  major  decision  point  in  the  program  to 
assure  that  the  major  objectives  of  that  phase 
have  been  met  before  starting  the  next  phase. 
However,  if  development  testing  is  not  complete 
before  the  first  major  production  decision,  the 
implementing  command  may  carry  on  additional 
DT&E  after  this  decision  if  it  is  required  for  de- 
velopment or  engineering  testing  of  an  updating 
change,  or  for  testing  a performance  area  not 
demonstrated  earlier  in  DT&E  (see  the  chart  in 
attachment  2). 

12.  What  Are  the  Major  Objectives  of  DT&E. 

This  phase  of  T&E  is  carried  out  in  order  to  do 
the  following: 

a.  Assess  the  critical  questions  and  areas  of  risk 
of  the  system,  and  carry  out  the  development  ob- 
jectives specified  in  the  program  documents.  In 
the  process: 

(1)  Identify  deficiencies  in  the  system,  and 
determine  the  degree  to  which  the  development 
contract  specifications  have  been  met. 

(2)  Insure  the  compatibility  and  performance 
of  the  support  items  (for  example,  simulators, 
life-support  systems,  support  equipment,  compu- 
ter resources,  technical  manuals,  and  other  data). 

(3)  Provide  estimates  of  system  reliability 
and  maintainability  to  be  expected  when  de- 
ployed. 

(4)  Determine  whether  the  system  is  safe  and 
ready  for  operational  testing. 

b.  Provide  data  with  which  to  assess  technical 
risk  and  evaluate  trade-offs  among  specifications, 
re(iuirements,  life  cycle  cost  and  schedules.  In 
addition: 

(1)  Accumulate  and  provide  data  for  the  es- 
timation of  survivability,  vulnerability,  and  logis- 
tic support  of  the  system. 

(2)  Provide  data  for  refining  estimates  of  re- 
quirements for  training  programs  aiid  training 
equipment. 

(3)  Provide  information  on  environmental  is- 
sues to  be  used  in  preparing  impact  assessments. 

c.  Insure  design  integrity  over  the  specified 
operational  and  environmental  range  by  conduct- 
ing preproduction  qualification  tests. 

13.  Policy  on  Managing  the  DT&E  Program: 

a.  The  implementing  command  is  responsible  for 
DT&E  program  management.  AFTEC  and  the 


MAJCOMs  will  support  DT&E  as  specified  in  pro- 
gram directives  and  in  the  planning  documents. 
The  implementing  command  develops  these 
documents  in  conjunction  with  AFTEC  and  the 
MAJCOMs. 

b.  The  contractor,  under  the  direction  of  the 
implementing  command,  usually  conducts  the 
early  part  of  DT&E,  which  includes  the  prepro- 
duction qualification  tests. 

(1)  The  implementing  command  (with  sup- 
port from  AFTEC  and  the  MAJCOMs  specified  in 
the  planning  documents),  will  take  part  in  the 
contractor-conducted  DT&E,  to  assist  in  plan- 
ning and  conducting  the  testing;  to  provide  visi- 
bility of  potential  problem  areas  and  an  early  as- 
sessment of  system  capabilities;  and  to  minimize 
duplication  of  Air  Force  and  contractor  testing. 

(2)  The  Air  Force  will  assume  complete  re- 
sponsibility for  the  conduct  of  DT&E  at  the  ear- 
liest practical  point. 

(3)  Air  Force  Preliminary  Evaluations 
(AFPEs)  may  be  conducted  to  determine  the  ex- 
tent that  the  DT&E  objectives  prescribed  in  the 
program  documents  have  been  met,  or  discrepan- 
cies have  been  resolved. 


14.  Concepts  for  Operational  Testing  and 
Evaluation.  This  phase  is  generally  divided  into 
two  types  of  OT&E:  initial  OT&E  (lOT&E)  and 
follow-on  OT&E  (FOT&E). 

a.  In  OT&E,  the  operational  environment  must 
be  as  realistic  as  possible,  to  assure  that  system 
performance  and  supportability  criteria  can  be 
evaluated  under  conditions  similar  to  those  in  the 
intended  operational  environment.  Con- 
sequently, all  OT&E  objectives  should  be 
examined  as  thoroughly  as  feasible  during 
lOT&E.  However,  since  the  estimate  of  the  sys- 
tem’s military  utility  (including  the  identification 
of  operational  deficiencies)  is  a primary  factor  in 
the  first  major  production  decision,  it  should  be 
thorough  enough  to  provide  a reasonable  assess- 
ment of  the  system’s  military  utility  in  its  in- 
tended operational  environment.  Therefore: 

(1)  lOT&E  will  begin  as  early  as  possible  in 
the  system  life  cycle,  using  prototypes  (prepro- 
duction items  or  RDT&E-funded  pilot  production 
items)  as  test  vehicles  (if  they  are  reasonably 
representative  of  the  expected  production  items) 
and  will  end  with  the  first  major  production  deci- 
sion. 

(2)  When  major  modification  is  required  be- 
cause of  future  system  applications,  threats,  tac- 
tics, or  doctrine,  the  modification  program  may, 
in  itself,  be  considered  as  a new  system  develop- 
ment program,  and  may  undergo  lOT&E  to  sup- 
port a production  decision  (see  attachment  2). 
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b.  FOT&E  is  conducted  to  continue  to  refine 
estimates  of  the  system’s  military  utility,  to  as- 
sist in  further  production  decisions  or  system  con- 
figiiration  changes  that  are  necessary. 

(1)  F’OT&E  will  begin  after  the  first  major 
production  decision,  and  may  continue  through- 
out the  system’s  life  cycle. 

(2)  Pilot  or  preproduction  test  items  will  be 
used  for  FOT&E  if  production  items  are  not  yet 
available. 

15.  What  Are  the  Major  Objectives  of  OT&E. 

The  objectives  of  this  phase  of  T&E  are  as  fol- 
lows: 

a.  To  estimate  the  operational  effectiveness  and 
operational  suitability  of  the  system,  as  well  as 
other  operational  aspects  of  its  military  utility. 

b.  To  identify  any  operational  deficiencies. 

c.  To  recommend  and  evaluate  desirable 
changes  and  trade  offs  in  production  configura- 
tion. 

d.  To  provide  information  for  use  in  refining 
training  concepts  and  programs;  in  developing 
and  refining  tactics,  techniques  and  doctrine;  and 
in  supporting  and  updating  technical  orders  and 
other  publications. 

e.  To  provide  information  obtained  during  ac- 
tual use  to  permit 

(1)  Refinement  of  official  program  operating 
and  support  (O&S)  cost  estimates,  and 

(2)  Identification  of  system  characteristics  or 
deficiencies  which  significantly  impact  O&S 
costs. 

16.  Policy  on  the  Conduct  of  OT&E  Programs. 

AFTEC  is  responsible  for  Air  Force  OT&E  pro- 
gram management;  responsibility  for  conducting 
OT&E  is  assigned  to  AFTEC  or  the  MAJCOMs 
as  follows: 

a.  AFTEC  will  conduct  OT&E  on  major  sys- 
tems and  on  those  nonmajor  systems  designated 
by  HQ  USAF.  On  these  OT&E  programs,  the 
AFTEC  commander  appoints  the  OT&E  test  di- 
rector from  the  AFTEC  staff  or  from  the  MAJ- 
COM  (with  the  concurrence  of  the  MAJCOM 
commander).  This  test  director  e.xercises  opera- 
tional control  over  the  OT&E  team.  Here,  “oper- 
ational control”  includes  decision  authority  over 
the  use  or  movement  of  all  assigned  OT&E  team 
members  and  OT&E  resources  identified  in  the 
test  program  documentation. 

b.  MAJCOMs  will  conduct  OT&E  on  all  other 
nonmajor  systems  as  designated  by  HQ  USAF.  A 
MAJCOM  commander  may  also  initiate  an  OT&E 
program  to  satisfy  command  requirements.  The 
MAJCOMs  will  establish  a command  focal  point 
to  serve  as  the  point  of  contact  for  OT&E  mat- 
ters. 


c.  AFTEC  will  monitor  nonmajor  lOT&E  pro- 
grams that  are  managed  by  MAJCOMS  and  other 
selected  OT&E  programs. 

d.  AFTEC  (or  the  MAJCOM  designated  to 
conduct  OT&E)  will  inform  the  program  manager 
and  the  using  (operating)  and  supporting  com- 
mands of  deficiences  as  soon  as  they  are  iden- 
tified. The  implementing  command  will  initiate 
action  to  resolve  the  deficiencies  in  coordination 
with  AFTEC  and  the  MAJCOMs. 

e.  ICBM  operational  test.  Phase  I and  II,  are 
directed  by  the  Joint  Chiefs  of  Staff  and  are  con- 
ducted to  determine  (Phase  I)  or  verify  (Phase  II) 
the  weapon  system’s  reliability  and  accuracy  fac- 
tors for  the  Single  Integrated  Operational  Plan 
(SIOP). 


17.  Conducting  a Combined  Test  Program: 

a.  Operational  testing  should  be  separate  from 
development  testing.  However,  when  separate 
testing  would  cause  a delay  involving  unaccepta- 
ble military  risk  or  cause  an  unacceptable  in- 
crease in  the  acquisition  cost,  combined  testing 
should  be  considered  early  in  the  program  plan- 
ning. The  planning  should  assure  that  combined 
testing  would  fulfill  developmental  and  opera- 
tional test  objectives.  If  the  test  program  is  com- 
bined: 

(1)  The  implementing  command  will  prepare 
the  DT&E  plan  and  be  responsible  for  integrating 
OT&E  requirements  into  a combined  test  pro- 
gram. 

(2)  AFTEC  (or  the  designated  MAJCOM) 
will  prepare  the  OT&E  plan  and  forward  it  to  the 
implementing  command  for  incorporation  into  the 
combined  program  documentation. 

(3)  AFTEC  (or  the  designated  MAJCOM) 
will  coordinate  on  the  combined  program 
documentation  to  insure  that  it  includes  ()T&E 
requirements. 

b.  In  conducting  the  combined  test  program, 
the  implementing  command  provides  a test  direc- 
tor who  is  responsible  for  DT&E  , for  integrating 
combined  test  events,  and  for  insuring  that  re- 
sources are  made  available  to  carry  out  the  com- 
bined test  program.  In  addition: 

(1)  AFTEC  (or  the  designated  MAJCOM) 
will  provide  the  OT&E  test  director  who  manages 
the  OT&E  portion  of  the  combined  program.  The 
resources  under  the  operational  )ntrol  of  this 
test  director  will  also  be  made  available  to  sup- 
port the  DT&E  test  plan,  as  specified  in  the  com- 
bined test  documentation. 

(2)  The  test  director  and  OT&E  test  director 
will  make  available  additional  support  under  their 
operational  control  as  required  and  as  agreed  ‘ 
upon  mutually. 
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c.  OT&E  personnel  will  participate  actively  in 
the  combined  test  to  enable  the  successful  accom- 
plishment of  OT&E.  To  avoid  duplication  of  test- 
ing, all  test  data  will  be  made  available  to  par- 
ticipating test  agencies,  as  required  to  accom- 
plish T&E  objectives. 

d.  To  assure  an  independent  operational  as- 
sessment under  combined  testing,  the  combined 
program  documentation  may  specify  that  AFTEC 
or  the  MAJCOM  conduct  separate  test  events 
when  AFTEC  (or  the  MAJCOM)  determines  that 
the  system’s  operational  capabilities  cannot  be 
properly  assessed  under  the  combined  testing. 
However,  the  test  director  from  the  implement- 
ing command  will  review  the  safety  aspects  of  the 
program. 

e.  AFTEC  (or  the  designated  MAJCOM)  will 
provide  to  the  CSAF  a separate  operational 
evaluation  of  the  resultant  test  information. 

f.  The  implementing  command  test  director  of 
the  combined  test  program  is  responsible  for  the 
safe  conduct  of  all  tests.  He  will  have  approval 
authority  for  all  DT&E  test  events  and  approval 
authority  to  insure  that  OT&E  events  comply 
with  safety  standards. 

SECTION  D— ASSIGNMENT  OF  RESPON- 
SIBILITIES 

18.  Responsibilities  of  H<^  USAF: 

a.  Publish  the  formal  Air  Force  programming 
and  management  documentation. 

(1)  Provide  test  and  evaluation  directives  via 
PMDs  (including  the  TEOA)  and  test  directives 
(TDs);  also,  assign  precedence  ratings  to  all  HQ 
USAF-directed  test  and  evaluation  projects. 

(2)  Establish  and  publish  in  appropriate 
documents  the  initial  critical  questions  and  areas 
of  risk  subsequently  identified  and  refined  by  the 
implementing  command,  AFTEC,  and  the  par- 
ticipating commands. 

(3)  Designate  the  agencies  or  commands  to  be 
responsible  for  specific  test  and  evaluation  pro- 
grams, including  the  extent  of  AFTEC  and 
MAJCOM  participation  in  FOT&E. 

(4)  Review  (and  approve,  when  specified) 
system  and  project  documents. 

(5)  Review  test  requirements  and  approve 
the  allocation  of  HQ  USAF-controlled  re.sources; 
also  provide  in.structions  for  the  disposition  and 
support  of  test  articles  in  the  PMD  before  the 
production  decision  (DSARC  III  on  a major  pro- 
gram). 

(6)  Approve  the  OT&E  Master  Program. 

(7)  Provide  to  AFTEC  (or  the  designated 
MAJCOM)  the  O&S  cost  estimates  for  each  new 
.system  as  estimates  are  developed  and  refined. 
This  includes  adequate  detail  on  the  methodology 


and  assumptions  to  permit  the  OT&h2  team  to 
construct  appropriate  cost  elements. 

b.  Provide  an  office  of  primary  responsibility 
for  OT&E  matters  and  keep  CS.AF  fully  informed 
as  to  the  needs  and  accomplishments  of  OT&E. 

c.  Resolve  any  intercommand  differences  which 
may  exist  concerning  T&E. 

d.  Provide  draft  T&E  directives  to  AFTEC  or 
the  appropriate  .MA.ICOMs  for  comment. 

e.  Implement  follow-up  action  on  test  fimlings, 
test  reports,  and  evaluation  recommendations. 

f.  Approve  the  programming  of  Air  F'orce  test- 
ing in  support  of  other  government  agencies  (as 
appropriate)  and  nongovernment  agencies. 

19.  Responsibilities  of  the  Implementing 
Command.  This  command  (usually  AFSC)  will: 

a.  Perform  planning  and  programming  for 
specified  acquisition  programs  and  projects. 

(1)  Plan,  program  and  budget  (according  to 
AFM  172-1)  for  test  resources  for  T&E.  Prepare 
and  coordinate  T&E  portions  of  appropriation 
program  documents  with  all  agencies  having  pro- 
gram responsibilities. 

(2)  Designate  the  responsible  development 
test  agencies  for  DT&E  in  the  PMP  and  related 
program  documentation. 

(3)  In  coordination  with  AFTEC  or  the  appro- 
priate MAJCOM,  establish  significant  test  miles- 
tones to  be  included  in  the  PMP,  TEMP,  and  re- 
lated program  documentation. 

(4)  As  directed  by  the  PMD,  prepare  a draft 
TEOA,  in  coordination  with  AFTEC  and  the  par- 
ticipating MAJCOMs,  and  submit  it  to  HQ  USAF 
for  approval.  AP’TEC  and  the  participating 
MAJCOMs  will  also  provide  OT&E  objectives  to 
the  implementing  command. 

(a)  The  approved  TFIOA,  which  is  dissemi- 
nated as  an  annex  to  the  PMD,  will  list  the 
specific  objectives  of  the  T&E  program.  These 
objectives  will  be  designed  to  provide  answers  to 
the  critical  questions  and  areas  of  risk  stated  in 
the  DCP  or  PM. 

(b)  The  annex  then  becomes  the  baseline 
for  test  data  evaluations. 

(5)  Provide  for  contractor  participation  in 
test  and  evaluation  as  required. 

(6)  In  coordination  with  the  participating  test 
agencies,  develop  T&  E alternatives  early  in  pro- 
gram planning  to  determine  whether  separate  or 
combined  testing  will  save  time,  money,  and  in- 
sure adequacy  of  testing.  When  directed,  submit 
recommendations  to  HQ  USAF  for  review  and 
approval. 

(7)  Insure  that  AFTEC  and  the  appropriate 
MA-JCOMs  have  an  early  opportunity  to  study 
and  comment  on  DT&E  of  .systems  identified  for 
pos.sible  acquisition. 
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(8)  Program  for  Joint  Test  Assembly  (JTA) 
nuclear  weapons  and  warheads,  as  specified  by 
the  MAJCOM. 

b.  Exercise  final  responsibility  for: 

(1)  DT&E  in  the  technology  base  program. 

(2)  DT&E  in  all  phases  of  the  system  life  cy- 
cle. 

(8)  Implementing  combined  test  programs. 

c.  In  conjunction  with  AFTEC  and  MAJCOMs, 
identify  the  critical  questions  and  areas  of  risk  to 
be  addressed  as  test  objectives  during  T&E,  and 
develop  and  plan  the  T&E  program  to  meet  the 
program  decision  milestones  (DSARC  milestones 
for  major  programs). 

d.  Insure  that  T&E  planning  includes  prepara- 
tion for  logistics  support  of  test  articles,  and  as- 
sures adequate  logistics  support  for  all  phases  of 
T&E  throughout  the  entire  effort,  to  include  re- 
tention as  a test  article  for  other  testing  after 
program  completion. 

e.  As  early  as  possible  before  starting  full-scale 
engineering  development,  prepare  the  TEMP, 
detailed  test  plans,  and  other  program  docu- 
ments; after  coordinating  these  plans  and  docu- 
ments with  AFTEC  and  appropriate  MAJCOMs, 
identify  and  integrate  the  effort  and  schedules  of 
all  T&E,  and,  in  conjunction  with  AFTEC  (or 
designated  MAJCOM),  insure  that  all  necessary 
T&  E is  completed  before  the  key  decision  points; 
keep  the  TEMP  current;  and  show  the  rationale 
and  approval  authority  for  each  change  in  it. 

f.  Plan  and  budget  for  required  update  or  mod- 
ification of  test  articles  based  upon  HQ  USAF  di- 
rection. 

g.  In  coordination  with  AFTEC  or  the  appro- 
priate MAJCOMs,  develop  and  implement  an  ef- 
fective data  system,  analytical  techniques,  and 
test  procedures  to  assess  design  reliability  and 
maintainability  and  logistic  support  models. 

(1)  Collect  and  process  reliability  and  main- 
tainability data  (see  AFR  80-5)  and  provide  it  to 
appropriate  agencies  responsible  for  evaluating 
reliability,  maintainability,  availability,  logistics 
supportability,  and  O&S  cost  estimates. 

(2)  Collect,  analyze,  and  evaluate  test  data 
and  prepare  and  distribute  reports  on  develop- 
mental tests. 

(3)  Review  test  results  & reported  deficien- 
cies to  recommend  corrective  action. 

h.  Provide  representation  and  furnish  en- 
gineering and  test  support  during  OT&E  until 
AFLC  assumes  program  management  responsi- 
bility for  the  system. 

20.  Responsibilities  of  the  Air  Force  Test  and 
Evaluation  Center.  AFTEC,  as  the  manager  of 
the  Air  Force  OT&E  program  will  (in  addition  to 
its  responsibilities  assigned  under  AFR  23-36): 


a.  Plan,  direct,  conduct,  control  and  independ- 
ently evaluate  the  report  to  CSAF  on  major  and 
HQ  USAF-designated  non-major  OT&E  pro- 
grams through  at  least  first  production  article 
testing. 

b.  In  consonance  with  their  mission  respon- 
sibilities, insure  that  Air  h'orce  OT&E  is  effec- 
tively planned  and  conducted. 

c.  For  HQ  USAF-designated,  MA.ICOM- 
conducted  OT&E  programs,  AFTEC  will  ap- 
prove the  test  plans  and  comment  on  the  final  re- 
port, if  appropriate.  For  lOT&E,  AF'TEC  will 
participate  with  sufficient  activity  and  program 
involvement  to  permit  a thorough  evaluation  of 
the  program.  AFTEC  will  submit  an  evaluation 
of  the  final  report  to  the  CSAF,  with  information 
copies  to  the  appropriate  MAJCOM,  the  program 
manager,  the  HQ  USAh’  OPR  for  OT&E,  and 
other  interested  agencies. 

d.  For  MAJCOM-initiated  programs,  comment 
as  appropriate  on  MAJCOM  test  plans  and  evalu- 
ation reports  and  include  appropriate  information 
in  the  OT&E  data  file  (see  below,  subpara  j). 

e.  Serve  as  the  principal  field  command  for 
providing  OT&E  information  to  the  Secretary  of 
Air  Force  (SAF)  and  CSAF  in  preparation  for 
DSARC  or  Air  Force  decision  milestone  reviews 
for  major  programs,  other  program  reviews,  and 

in  support  of  Air  Force  procurement  reciuests  for  ^ 
which  OT&E  information  is  statutorily  required 
to  be  supplied  to  the  Congress  (through  HQ 
USAF). 

f.  Submit  for  HQ  USAF  approval  the  OT&E 
portions  of  DCPs,  PMs,  PMDs,  and  other  appro- 
priate documents  with  inputs  from  the  MAJ- 
COMs. 

g.  Accomplish  detailed  planning  and  budgeting 
for  OT&E  in  accordance  with  AFM  172-1.  Furnish 
the  implementing  command  and  other  participat- 
ing commands  copies  of  the  detailed  planning  in 
sufficient  time  to  permit  inclusion  of  support  re- 
quirements in  their  budget. 

h.  Participate  with  the  implementing  command 
in  preparation  of  the  TEMP/TEOA,  prepare  the 
OT&E  portion  of  the  TEMP/TEOA,  and  coordi- 
nate on  the  TEMP/TEOA. 

i.  Provide  information  on  deficiencies,  as  they 
occur,  to  the  program  manager  and  to  affected 
MAJCOMs. 

j.  Maintain  an  OT&E  Master  Program,  Master 
Instrumentation/Capabilities  Plan/Guide,  and  a 
data  file  on  Air  Force  OT&E  programs,  as  appro- 
priate. 

k.  Provide  copies  of  planning  documents, 
interim  reports,  and  final  reports  of  electronic 
warfare — related  T&E  (including  radar  cross  sec- 
tion tests)  to  the  USAFSS/Air  Force  Electronic 
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Warfare  Center/SUI)  for  inclusion  in  the  T&E 
USAP'  electronic  warfare  (EW)  data  bank. 

l.  Provide  operating  and  support  (O&S)  cost  in- 
formation resulting  from  operational  suitability 
testing  to  agencies  responsible  for  developing, 
validating,  or  updating  the  inputs  and  cost  factors 
u.sed  in  O&S  cost  models.  This  information  will 
emphasize  the  test  sensitive  elements  of  O&S 
cost  estimates. 

m.  Identify  to  HQ  USAF  those  modification 
programs  that  will  retiuire  the  satisfaction  of 
OT&E  objectives  beyond  normal  modification 
testing  or  proofing. 

21.  Responsibilities  of  the  .Major  Commands. 
Each  .MAJCOM  will  maintain  a command  focal 
point  for  T&E  and  perform  these  functions,  as 
appropriate: 

a.  Provide  operational  employment  and 
maintenance  concepts  for  T&E  program  use  in 
accordance  with  paragraph  5b  above. 

b.  Identify  critical  questions  and  areas  of  risk 
(within  their  cognizance)  for  DT&E  and  OT&E 
for  submission  to  the  responsible  test  organiza- 
tions. 

b.  Participate  early  in  the  planning  and  pro- 
gramming of  test  and  evaluations.  Provide 
budget  requirements,  system  test  requirements 
(to  include  training  required  for  operational  per- 
sonnel), and  test  article  requirements  for  OT&E 
to  AFTEC  on  a schedule  to  support  established 
budget  and  RF’P  cycles.  Fund  for  resources  as 
prescribed  in  AFM  172-1.  Participate  in  prepara- 
tion of  the  TEMP/TEOA  and  in  T&E  as  directed 
by  program  documentation  (for  e.xample,  PMDs, 
PMPs,  and  HQ  USAF  TDs). 

(1)  Review  and  coordinate  on  test  plans  and, 
as  specified  (in  PMDs,  TDs,  or  approved  test 
plans),  provide  qualified  personnel  to  the  test 
teams  and  provide  resources  to  other  commands 
or  agencies  for  the  T&E  projects. 

(2)  In  coordination  with  AFTEC  and  the  im- 
plementing command,  plan  and  program  required 
manpower  spaces  to  insure  that  qualified  person- 
nel are  available  for  T&E. 

d.  Provide  data  on  the  adequacy  of  established 
requirements  for  manpower  and  training  to  oper- 
ate and  maintain  the  system  in  its  operational  en- 
vironment, with  recommendations  for  establish- 
ing or  changing  Air  Force  specialties.  Also,  de- 
termine the  adequacy  of: 

(1)  Technical  data  available  to  support  and 
operate  the  system. 

(2)  Health  protection,  life  support,  and  medi- 
cal safety  procedures,  directives,  and  equipment. 

(3)  Support  equipment  (including  materials 
handling  equipment). 


(4)  Facilities  programmed  and  uniiiue  or  addi- 
tional reijuiremetits  essential  to  successful 
employment. 

e.  Collect  and  process  data,  using  standard  .Air 
Force  data  systems  or  command-uni(|Ue  .systems, 
to  as.sess  reliability,  maintainability,  availability, 
logistics  supportability,  and  O&S  cost  impacts. 
Standard  or  specially  developed  data  retrieval 
and  analysis  programs  will  be  used  for  the  as- 
sessments. 

(1)  Develop  and  provide  to  AFTFX’  (or  the 
designated  MAJCOM)  the  models,  techni(iues  or 
cost  element  data  used  to  construct  estimates  of 
operating  and  support  costs.  These  are  to  be  u.sed 
during  OT&E  to  actiuire  information  which  can  be 
used  to  improve  and  validate  O&S  cost  estimates 
and  identify  system  characteristics  or  deficien- 
cies which  impact  O&S  costs. 

(2)  Develop  and  establish  j)rocedures  to  bring 
about  a continuing  interchange  of  information 
with  other  commands  or  agencies. 

(8)  Provide  technical  information  reciuested 
by  the  testing  activity  for  the  planning  and  con- 
duct of  testing. 

(4)  Provide  copies  of  planning  documents, 
interim  reports,  and  final  reports  of  electronic 
warfare  (EW)  related  T&E  (including  radar  cross 
section  tests)  to  the  USAFSS/Air  F’orce  Elec- 
tronic Warfare  Center/SUD  for  inclusion  in  the 
T&E  USAF  EW  data  bank. 

f.  Review  test  reports  and  recommendations, 
and  implement  actions  on  test  findings  falling 
within  command  cognizance. 

22.  Responsibilities  of  the  Using  Commands.  In 
addition  to  the  responsibilities  in  paragraph  21, 
the  using  commands  will: 

a.  Accomplish  HQ  USAF’-designated  (or 
command-initiated)  OT&E,  with  support  from 
the  implementing  command  or  participating 
MA,ICOMs. 

(1)  For  HQ  USAF'-designated  OT&E,  obtain 
AFTEC  approval  on  test  plans. 

(2)  Forward  reports  on  OT&E  to  AFTEC  for 
evaluation,  comment  and  submission  to  CSAF,  as 
specified  in  paragraph  26c. 

(8)  For  command-initiated  OT&E,  prepare 
and  coordinate  plans  with  all  participating  MAJ- 
COMs;  forward  test  plans  and  reports  to  AFTEC 
for  review  and  comment. 

(4)  For  HQ  USAF-designated  OT&E,  pro- 
vide a Test  Director;  also,  for  AF'TEC-conducted 
OT&E,  when  requested  by  the  AFTEC  com- 
mander. 

(5)  For  HQ  USAF-designated  OT&E,  pro- 
vide O&S  cost  information  resulting  from  opera- 
tional suitability  testing  to  responsible  agencies 
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for  updatinjj  official  program  O&S  cost  estimates. 
Emphasis  will  be  on  the  elements  of  these  esti- 
mates which  are  test  sensitive. 

b.  Plan,  program,  budget  (according  to  AFM 
172-1),  and  provide,  from  within  the  command, 
the  resources  (personnel,  equipment,  flying 
hours,  operating  budgets,  and  so  forth)  necessary 
to  accomplish  OT&E. 

c.  Manage  that  OT&E  which  primarily  con- 
cerns operational  training;  employment  exer- 
cises, tests  undertaken  to  develop  or  refine  tac- 
tics, techniques,  procedures,  and  doctrine;  or 
tests  and  exercises  to  establish  or  verify  system 
reliability  and  accuracy  factors  for  deployed 
strategic  forces  as  required  by  the  JCS  for  the 

Slop. 


23.  Responsibilities  of  the  Air  Force  Logistics 
Command.  In  addition  to  the  responsibilities  in 
paragraph  21,  AFLC  will: 

a.  Provide  logistic  support  and  planning  for 
test  programs. 

b.  Identify,  to  the  test  organization,  the  test 
objectives  and  data  requirements  in  logistic  areas 
for  new  systems  under  T&E.  Define  maintenance 
test  objectives  in  relation  to  the  initial  collection 
and  preparation  of  equipment  allowances  list  of 
tools,  test  sets,  and  calibration  equipment  re- 
quirements. 

c.  In  coordination  with  AFTEC,  identify  to  HQ 
USAF  those  modification  programs  that  may  re- 
quire the  satisfaction  of  OT&E  objectives  beyond 
normal  modification  test  or  proofing. 

d.  When  designated  as  a supporting  command, 
participate  on  test  teams  for  planning,  conduct- 
ing, and  reporting  determination  of  logistics  sup- 
portability  for  DT&E,  and  verify  or  assess  these 
factors  for  OT&E. 

(1)  During  MAJCOM-conducted  OT&E,  par- 
ticipate on  the  test  team  as  necessary  to  assess 
reliability,  maintainability,  availability,  suppor- 
tability  and,  when  applicable,  O&S  cost  impacts. 

(2)  When  DT&E  and  OT&E  programs  are 
separate,  provide  a special  assistant  for  inte- 
grated logistics  to  the  test  director  (from  the  im- 
plementing command)  during  DT&E,  and  to  the 
OT&E  test  director,  to  assist  in  making  a valid 
assessment  of  logistics  supportability. 

(3)  For  an  AFTEC-conducted  OT&E  pro- 
gram, provide  membership  on,  and  a special  as- 
sistant for  integrated  logistics  to  chair,  the 
Logistics  Supportability  Evaluation  Team 
(LSET).  This  Team  is  responsible  for  assessing 
logistics  supportability  and  to  the  extent  testing 
permits,  the  validity  of  operating  and  support 
cost  estimates. 

(4)  In  a combined  test  program,  the  LSET 


will  assist  in  DT&E  and  OT&E,  as  mutually 
agreed  upon  by  both  test  directors. 

e.  Assist  AFTEC  (or  if  appropriate,  the  MAJ- 
COM),  in  developing  definitions,  criteria,  and 
computational  procedures  for  use  in  making 
OT&E  assessments  of  reliability,  maintainability, 
availability,  supportability,  and  O&S  cost  im- 
pacts. Through  evaluating  the  developmental,  op- 
erational, and  related  test  and  management  data, 
provide  input  to  test  reports  on  logistics  suppor- 
tability, and  system  effectiveness  and  suitability. 

f.  Pai'ticipate  with  th«  test  organization  in  a.s- 
suring  that  failure  rate  data  are  directed  to  all 
appropriate  agencies  by  priority  procedures; 
also; 

(1)  Incorporate  the  resultant  changes  into 
logistics  planning  for  the  operational  inventory  as 
early  as  practicable. 

(2)  Insure  that  procedures  are  established  for 
timely  feedback  of  T&E-generated  reliability  and 
maintainability  (R&M)  data  into  the  main- 
tenance-planning and  provisioning  procedures. 

g.  Provide  engineering  support  (including  Air 
Force  and  contractor  personnel)  during  OT&E 
•when  assigned  program  management  responsibil- 
ity. 

h.  Perform  the  functions  of  paragraph  19  when 
designated  as  the  implementing  command. 


24.  Responsibilities  of  the  USAF  Security  Serv- 
ice. This  agency  will: 

a.  For  SIGINT  Systems; 

(1)  Developed  by  the  National  Security 
Agency  (NSA)  for  multiservice  (including  Air 
Force)  use; 

(a)  Monitor  NSA  DT&E  and  plan,  pro- 
gram, and  provide  for  Air  Force  participation 
during  the  NSA  planning  and  conduct  the  OT&E. 

(b)  Prepare  comments  on  the  NSA  opera- 
tional test  reports.  These  comments,  along  with  a 
copy  of  the  NSA  report,  should  evaluate  the 
adequacy  of  NSA  testing,  discuss  any  disagree- 
ments with  the  conclusions  and  recommenda- 
tions, and  describe  the  action  taken  to  resolve 
any  differences.  Forward  these  comments  to 
CSAF  through  AFTEC. 

(2)  Developed  by  NSA  for  sole  use  by  Air 
Force: 

(a)  Monitor  NSA  DT&E. 

(b)  Plan,  program,  conduct,  and  report  on 
OT&E.  Forward  OT&E  report  to  NSA  and 
AFTEC  with  copies  to  other  interested  MAJ- 
COMs  and  agencies. 

(3)  Conduct  lOT&E  or  lOT&E  portions  of 
combined  tests  of  systems  developed  by  AFSC. 

b.  For  COMSEC  Equipment. 
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(1)  Monitor  and  participate  in  NSA- 
conducted  development  tests  of  NSA-developed 
COMSKC  equipment  proposed  for  Air  Force  use. 

(2)  For  NSA-developed  COMSEC  equip- 
ment. 

(a)  Provide  for  Air  Force  participation  in 
joint  -service  DT&E,  lOT&E,  and  combined  test 
programs  by  planning,  programming  and  coor- 
dinating with  AFTFX  and  the  participating 
MAJCOMs-SOAs. 

(b)  Conduct  DT&E  and  furnish  technical 
advice  and  assistance  to  AFTEC  (or  the  using 
commands,  as  appropriate)  in  the  conduct  of 
OT&E  of  systems  intended  for  Air  Force  use,  in- 
cluding the  Air  Force  portion  of  joint  service 
tests. 

(c)  Acquire  all  COMSEC  equipment  re- 
(juired  for  Air  Force  testing. 

(d)  Evaluate  and  program  logistic  support 
during  NSA  development  testing,  DT&E  and 
lOT&E,  to  provide  for  continuing  logistic  support 
of  equipment  owned  by  the  Air  Force. 

(e)  Coordinate  with  NSA  on  changes  re- 
quired as  a result  of  deficiencies  identified  during 
Air  Force  testing. 

(3)  Participate  with  AFSC  in  DT&E  of  com- 
bined test  programs  for  equipment  development 
and  production  delegated  to  the  Air  Force  as  out- 
lined in  AFM  100-45;  conduct  the  lOT&E  for  such 
systems.  Service  tests  of  NSA-developed  COM- 
SEC equipment  are  considered  combined  test 
programs.  Here,  NSA  uses  the  term  “service 
test"  for  testing  that  the  military  services  per- 
form to  verify  compliance  with  their  require- 
ments before  the  major  production  decision. 

c.  Conduct  all  required  Air  Force  TEMPEST 
testing  of  classified  information  processing  sys- 
tems acquired  for  Air  Force  use,  as  prescribed  by 
AFR  100-51. 

d.  Provide  electronic  warfare  (EW)  support  as 
follows; 

(1)  Establish  and  maintain  the  USAF  EW 
data  bank  according  to  AFR  55-90. 

(2)  Provide  EW  research  services  to  MAJ- 
COMs. 

(3)  Provide  technical  assistance  to  MAJ- 
COMs  in  planning,  conducting,  and  evaluating 
tests  of  EW  systems. 

25.  Responsibilities  of  the  Air  Training  Com- 
mand. In  addition  to  the  responsibilities  in  para- 
graph 21,  this  command  will: 

a.  Participate  in  system  acquisition  from  publi- 
cation of  the  formal  Air  Force  requirements 
document  through  DT&E  and  OT&E. 

b.  Prepare  a test  plan  to  support  AFTEC  and 
MAJCOMs  during  T&E,  provide  qualified  per- 
sonnel to  test  teams  as  required  by  PMD’s  or  Test 


directives,  and  compile  and  verify  the  training 
data,  training  programs,  curricula,  training 
standards,  and  activities. 

c.  Plan,  develop,  and  provide  adequate  and 
timely  training  for  personnel  to  test,  operate,  and 
maintain  equipment,  and,  when  appropriate,  ac- 
complish this  according  to  AFM  50-2,  In.struc- 
tional  System  Development. 

d.  Develop  tentative  specialty  training  stand- 
ards for  new  or  revised  Air  F orce  specialties  from 
evaluation  during  testing.  Include  the  evaluation 
of  these  standards  as  a test  objective  in  the  ATC 
portion  of  the  test  plan. 

SECTION  E— ADMINISTRATION 

26.  Reporting  Requirements: 

a.  T&E  Procedures.  Reporting  requirements 
are  specified  here,  in  AFR  23-36,  and  in  PMDs. 

(1)  Unless  otherwise  specified,  each  test  re- 
port is  to  be  released  for  publication  no  later  than 
60  calendar  days  after  the  final  test  event  is  com- 
pleted. Interim  reports  may  also  be  published. 
However: 

(a)  Each  test  report  must  first  be  reviewed 
for  the  proper  security  classification  and  assigned 
the  proper  distribution  statement  as  required  bv 
AFR  80^5. 

(b)  COMINT,  ELINT,  and  COMSEC  final 
test  reports  must  be  distributed  as  specified  in 
NSA  Circular  80-6  (2). 

(c)  Copies  of  all  other  reports  must  be  sent 
to  the  Defense  Documentation  Center,  Cameron 
Station,  Alexandria  VA  22314  (if  it  carries  a dis- 
tribution statement  A,  send  12  copies;  if  it  carries 
distribution  statement  B,  send  2 copies). 

(2)  If  the  testing  is  conducted  as  an  engineer- 
ing service,  the  participating  agencies  will  mutu- 
ally determine  the  reporting  requirements. 

(3)  When  ICBM  tests  are  conducted  at  the  di- 
rection of  JCS,  submit  reports  as  directed  in 
JCSM  478-75;  these  reports  are  controlled  and 
distributed  as  outlined  in  AFR  11-11  and  JCS  pol- 
icy memorandum  39. 

(4)  Test  reports  that  provide  evaluation  data 
or  information  relative  to  competitive  procure- 
ment will  be  handled  according  to  AFR  70-15. 

b.  DT&E  Procedures.  The  implementing  com- 
mand will  submit  DT&E  reports  to  the  HQ  USAF 
OPR,  with  information  copies  to  AFTEC  and  par- 
ticipating MAJCOMs. 

c.  OT&E  Procedures. 

(1)  On  each  AFTEC-conducted  OT&E  pro- 
gram, the  AFTEC  commander  will  forward  the 
OT&E  evaluation  report  directly  to  the  CSAF, 
with  information  copies  to  the  commanders  of  the 
participating  MAJCOMs,  HQ  USAF  OT&E  OPR, 
and  other  interested  agencies. 
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(2)  On  each  HQ  USAF-designated, 
MAJCOM-eonducted  OT&E  program,  the  MAJ- 
COM  will  forward  copies  of  the  evaluation  report 
to  the  CSAF  through  the  AFTEC  commander 
who  evaluates  and  comments  on  the  report,  if  ap- 
propriate. 

(a)  The  MAJCOM  will  forward  information 
copies  of  the  evaluation  report  concurrently  to 
the  HQ  USAF  OPR  for  OT&E,  and  to  other  in- 
terested agencies. 

(b)  The  AF'TEC  commander  will  forward 
comments  on  the  evaluation  report  to  the  CSAF, 
with  information  copies  to  the  HQ  USAF  OPR  for 
OT&E,  to  the  participating  MAJCOMs,  and  to 
other  interested  agencies. 

(3)  On  each  MAJCOM-initiated  OT&E  pro- 
gram, test  plans,  and  evaluation  reports  are  for- 
warded to  the  AFTEC  commander  for  comment, 
if  appropriate. 


(4)  During  the  conduct  of  OT&E,  any  system 
deficiency  will  be  reported  as  soon  as  possible  to 
the  implementing  and  participating  commands  in 
accordance  with  HQ  USAF  Air  Force  directives 
and  policies. 

27.  Authority  for  Direct  Communication.  Di- 
rect communication  is  authorized  between  Air 
Force  Commands  and  the  Army,  Navy,  Marine 
Corps,  and  other  Government  agencies  in  imple- 
menting this  regulation. 

28.  Document  Disposition.  Disposition  of 
documentation  prescribed  by  this  directive  will 
be  made  according  to  AFM  12-50. 

29.  Funding.  F'unding  for  T&E  will  be  in  accord- 
ance with  AFM  172-1. 


BY  ORDER  OF  THE  SECRETARY  OF  THE  AIR  FORCE 


OFFICIAL  DAVID  C.  JONES,  General,  USaF 

Chief  of  Staff 

JAMES,!.  SHEPARD,  Colonel,  USAF 
Director  of  Administration 

SUMMARY  OF  REVISED,  DELETED,  OR  ADDED  MATERIAL 

This  revision  clarifies  management  relationships  and  responsibilities  between  AFTEC,  the  implement- 
ing command,  and  other  commands  in  the  conduct  of  OT&E  and  combined  development/operational 
testing  (Sections  B & D);  stipulates  the  participation  early  in  system  development  of  OT&E  personnel 
(para  2a,  (3),  7,  14a(l)):  redefines  lOT&E  as  that  OT&E  conducted  prior  to  the  first  major  production 
decision,  and  FOT&E  as  all  OT&E  after  that  decision  (para  14);  addresses  the  conduct  of  ODDR&E 
sponsored  joint  Service  testing,  and  applies  the  principles  of  the  regulation  to  Foreign  Military  Sales, 
where  feasible  (para  2d  (5)  );  encompasses  T&E  in  simulator  and  system  modification  programs  (para 
2a(3)(d),  6b,  6c);  implements  the  requirement  to  investigate  the  appropriateness  of  multi-service  testing 
and  two-sided  testing  during  DCP  development  (para  5);  changes  the  title  of  Master  Instrumentation/ 
capabilities  Program  to  Master  Instrumentation/Capabilities  Plan-Guide  (para  20j);  limits  AFTEC’s 
monitorship  role  (para  20c,  22a(l)  and  26c(3);  and  adds  a format  for  the  Test  and  Evaluation  Objectives 
Annex  (atch  3). 
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1.  Acceptance  Tests.  Those  tests  performed  to 
demonstrate  that  a specific  lot  of  articles  have 
been  manufactured  to  specification  tolerances. 

2.  .Acquisition  Process.  Normally,  it  consists  of 
five  phases  (Conceptual,  Validation,  Full  Scale 
Engineering  Development,  Production  and  De- 
ployment) with  key  decision  points  after  each  of 
the  first  three  phases.  A program  may  skip  a 
phase,  or  may  have  program  elements  in  any  or  all 
phases.  (See  AFK  80()-2  for  complete  e.xplana- 
tion.) 

3.  Air  Force  Preliminary  Evaluation  (AFPE). 

The  AFPE  is  conducted  to  evaluate  the  system 
performance  and  the  technical  and  engineering 
potential,  identify  any  gross  deficiencies,  and  de- 
termine the  degree  to  which  contract  specifica- 
tions are  being  met  (will  include  an  assessment  of 
operational  effectiveness  and  suitability). 

4.  Availability.  Availability  is  a measure  of  the 
degi'ee  to  which  an  item  is  in  the  operable  and 
commitable  state  at  the  start  of  the  mission  when 
the  mission  is  called  for  at  an  unknown  (random) 
time  (inherent  availability)  (MIL-STD-721B). 
For  OT&E  purposes,  availability  is  considered 
synonymous  with  operational  readiness. 

5.  Compatibility.  The  capability  of  two  or  more 
operational  items/systems  to  exist  or  function  as 
elements  of  a larger  operational  system  or  opera- 
tional environmental  without  mutual  interfer- 
ence. 

6.  Defense  Systems  Acquisition  Review  Council 
(DSARC).  An  advisory  council  established  by  and 
functioning  for  the  Secretary  of  Defense  (SEC- 
DEF)  to  apprise  the  SECDEF  of  the  program 
status  and  readiness  of  a major  defense  system 
prior  to  proceeding  to  the  next  phase  in  the  acquis- 
ition process. 

7.  Evaluation.  The  review  and  analysis  of  qual- 
itative and/or  quantitative  data  obtained  from  de- 
sign review,  hardware  inspection,  testing,  and/or 
operational  usage  of  equipment. 

8.  First  Major  Production  Decision.  The  deci- 
sion to  begin  production  of  procurement-funded 
end  items  intended  for  Service  deployment. 

9.  Follow-on  OT&E.  That  test  and  evaluation 
which  is  conducted  after  lOT&E  to  continue  and 
refine  the  estimates  made  during  the  lOT&E,  to 
evaluate  changes,  and  to  reevaluate  the  system  to 


insure  that  it  continues  to  meet  operational  needs 
and  retain  its  effectiveness  in  a new  environment 
or  against  a new  threat. 

10.  Implementing  Command.  The  command  re- 
sponsible for  the  acquisition  and/or  modification 
of  the  system,  subsystem,  or  item  of  equipment. 

11.  Initial  Operational  Test  and  Evaluation 
(lOT&E).  That  portion  of  operational  test  and 
evaluation  conducted  prior  to  the  first  major  pro- 
duction decision. 

12.  Interoperability.  The  ability  of  systems,  units 
or  forces  to  provide  services  to,  and  accept  serv- 
ices from,  other  systems,  units,  or  forces,  and  to 
use  the  services  so  exchanged  to  enable  them  to 
operate  effectively  together. 

13.  Life  Cycle  Cost.  The  total  cost  of  an  item  or 
system  over  its  full  life.  It  includes  the  cost  of  de- 
velopment, production,  ownership  (operation, 
maintenance,  support,  etc.)  and,  where  applica- 
ble, disposal  (see  AFR  800-11). 

14.  Logistics  Supportability.  How'  well  the  com- 
posite of  support  considerations  necessary  to 
achieve  the  effective  and  economical  support  of  a 
system  or  equipment  for  its  life  cycle  meets 
stated  quantitative  and  qualitative  requirements. 
This  includes  integrated  logistics  support  (ILS) 
and  logistics  related  O&S  cost  considerations. 

15.  Logistics  Supportability  Evaluation  Team 
(LSET).  An  AFTEC  or  MAJCOM  OT&E  test 
team  activity  assigned  overall  responsibility  for 
preparing  a unified  assessment  of  logistics  sup- 
portability. The  LSET  is  normally  chaired  by  the 
AFLC  Special  Assistant  for  Integrated  Logistics 
and  includes  representation  from  the  MAJCOMs 
and  other  activities  having  logistics  interest. 

16.  Maintainability.  A characteristic  of  design 
and  installation  expressed  as  the  probability  that 
an  item  will  be  restored  to  a specified  condition 
within  a given  period  of  time  when  the  mainte- 
nance is  performed  using  prescribed  procedures 
and  resources.  System  maintainability  may  also 
be  expressed  in  such  terms  as  Mean-Time-to- 
Repair,  Maintenance  Manhours  per  Flying  Hour, 
or  Mean  Down-Time  (see  AFR  80-5). 

17.  Military  Utility.  A generic  term  used  to  de- 
scribe the  value  of  an  item  or  system  with  respect 
to  a current  concept  of  operation. 
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18.  Operational  Effectiveness.  How  well  the 
system  performs  its  intended  mission  in  its  in- 
tended environment,  exclusive  of  system  support 
considerations.  Survivability,  compatibility,  and 
interoperability  may  be  considerations  in  evaluat- 
ing effectiveness. 

19.  Operational  Suitability.  How  well  the  sys- 
tem performs  its  intended  mission  when  operated 
and  maintained  by  military  personnel  in  the  field. 
This  normally  includes  capability,  availability,  re- 
liability, maintainability,  logistics  supportability, 
training  requirements,  and  an  assessment  of 
operating  and  support  cost  characteristics. 

20.  Pilot  Production.  A limited  production  run  of 
a new  system  which  has  completed  engineering 
development  and  for  which  the  capability  to  mass 
produce  the  item  for  inventory  needs  to  be  dem- 
onstrated. 

21.  Preproduction  Article.  An  article  which  is  in 
final  form,  employs  standard  parts  (or  nonstand- 
ard parts  approved  by  the  agency  concerned),  and 
is  repre.sentative  of  final  equipment. 

22.  Program  Manager.  The  single  Air  Force  man- 
ager during  any  specific  phase  of  the  acquisition 
process  (Program  Manager  or  System  Manager/ 
Item  Manager). 

23.  Prototype.  First  full-scale  functional  form  of 
a new  system,  subsystem,  or  component,  on 
which  the  design  of  subsequent  production  items 
is  patterned. 

24.  Qualification  Tests.  Those  tests  that  verify 
the  design  and  manufacturing  process  and  thus 
provide  a baseline  for  subsequent  acceptance 
tests.  Qualification  testing  is  conducted  to  ac- 
complish two  .separate  functions: 

a.  Preproduction  Qualification  Tests.  A series  of 
formal  contractual  tests  are  conducted  to  insure 
design  integrity  over  the  specified  operational  and 
environmental  range.  The  test  should  be  con- 
ducted on  prototype  or  preproduction  items  fabri- 
cated to  the  proposed  production  design  specifica- 
tions and  drawings.  These  tests  are  a constraint  to 
production  release  on  programs  which  involve 
volume  acquisition.  The  preproduction  qualifica- 
tion tests  include  those  contractual  reliability  and 
maintainability  demonstration  tests  required 
prior  to  production  release. 

b.  Production  Qualification  and  Acceptance 
Tests.  A series  of  formal  contractual  tests  are 
conducted  to  insure  the  effectiveness  of  the  man- 
ufacturing process,  equipment,  and  procedures. 
The.se  tests  are  conducted  on  a sample  taken  at 
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random  from  the  first  production  lot,  and  are  re- 
peated if  the  process  is  changed  significantly  and 
when  a second  or  alternate  source  is  brought  on 
line. 

25.  Reliability: 

a.  Hardware  Reliability.  Hardware  reliability 
is  the  probability  that  a part,  component,  subas- 
sembly, assembly,  subsystem,  or  system  will  per- 
form for  a specified  interval  under  stated  condi- 
tions with  no  malfunction  or  degradation  that  re- 
quires corrective  maintenance  actions  (AFR 
80-5). 

b.  Operational  Reliability.  The  probability  that 
an  operationally  ready  system  will  perform  as  re- 
quired to  accomplish  its  intended  mission  or  func- 
tion as  planned  (AFR  80-5). 

26.  Supporting  Command.  A command  that  pro- 
vides direct  support  to  a system  or  test  program. 
Normally  refers  to  AFLC,  USAFSS,  and  ATC  in 
their  role  as  logistics  support  and  training  or- 
ganizations. 

27.  Survivability.  The  capability  of  a system  to 
avoid  or  withstand  a man-made  hostile  environ- 
ment without  suffering  an  abortive  impairment  of 
its  ability  to  accomplish  its  designated  mission. 

28.  T&E  Master  Plan  (TEMP).  This  is  an  overall 
test  and  evaluation  plan  designed  to  identify  and 
integrate  the  effort  and  schedules  of  all  T&E  to 
be  accomplished  in  connection  with  an  acquisition 
program. 

29.  Test  and  Evaluation  Objectives  Annexes 
(TEOA)  To  PMI).  The  TEOA  provides  a common 
baseline  for  the  independent  evaluations  by  the 
implementing  command  and  AFTEC  or 
operating/supporting  commands.  The  TEOA  de- 
lineates discrete  objectives  of  the  overall  test 
program.  The  attainment  of  these  objectives  will 
provide  the  answers  to  the  critical  questions  and 
areas  of  risk  presented  in  the  DCP. 

30.  Test.  Any  program  or  procedure  which  is  de- 
signed to  obtain,  verify,  or  provide  data  for  the 
evaluation  of:  research  and  development  (other 
than  laboratory  experiments);  progress  in  ac- 
complishing development  objectives;  or  perform- 
ance and  operational  capability  of  systems,  sub- 
.systems,  components,  and  equipment  items. 

31.  Test  Directive.  A HQ  USAF  document  which 
provides  direction  and  guidance  for  OT&E  for 
those  cases  not  covered  by  PMD. 

32.  Test  Director.  A person  assigned  to  conduct  a 
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test  in  accordance  with  the  test  plan,  and  who 
exercises  overall  responsibility  for  achieving  test 
plan  objectives. 

33.  Test  Plan.  A formal  document  which  provides 
the  complete  detailed  coordination  and  integrated 


plan  for  the  time  pha.sed  task  of  providing 
answers  and  solutions  to  the  critical  questions 
and  areas  of  risk  identified  in  the  DCP  and  other 
program  documentation.  It  must  also  list  the  re- 
sources required  to  conduct,  analyze,  and  report 
on  the  test. 


TEST  AND  EVALUATION  ACTIVITIES 
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TEST  AND  EVALUATION  OBJECTIVES  ANNEX  (TEOA) 

Explanation.  The  TEOA  is  intended  to  be  a brief  statement  of  specific  T&E  objectives  that  will  serve  as 
the  baseline  for  all  evaluations  of  DT&  E and  OT&  E programs.  In  order  to  insure  that  all  of  the  commands 
and  agencies  involved  are  directed  toward  the  same  objectives  in  their  evaluation,  the  TEOA  will  be 
employed  when  more  than  one  command  or  agency  is  testing  or  evaluating  a system.  Its  only  purpose  is  to 
furnish  a clear  and  concise  statement  of  the  T&E  objectives.  (Organizational  relationships  and  respon- 
sibilities are  outlined  in  the  PMD  or  Program  Management  Plan,  as  required,  but  not  in  the  TEOA.) 

Since  satisfaction  of  most  DT&E  and  lOT&E  objectives  is  requisite  to  the  first  major  production 
decision,  the  TEOA  should  delineate  which  objectives  must  be  accomplished  before  production  deci- 
sion, and  which  after.  At  the  time  the  TEOA  is  first  issued,  FOT&E  objectives  may  be  fully  developed. 
Subsequent  updating  of  the  TEOA  may  then  be  required  to  complete  the  statement  of  FOT&E  objec- 
tives. 

The  following  outline  serves  as  the  basic  format  for  the  TEOA: 

TEST  AND  EVALUATION  OBJECTIVES  ANNEX  (TEOA)  FOR  (Subject  System) 

I.  INTRODUCTION:  An  introductory  statement  should  firmly  establish  the  intent  of  the  TEOA;  for 
example,  “This  Test  and  Evaluation  Objectives  Annex  (TEOA)  establishes  the  specific  objectives  of  the 
overall  test  and  evaluation  program  in  terms  of  the  operational  and  engineering  criteria  for  judging  of 
the  system’s  performance.  It  will  be  the  common  baseline  for  independent  evaluation  by  all  participat- 
ing organizations.  Achievement  of  the  objectives  in  this  TEOA  will  provide  answers  to  the  critical 
questions  and  areas  of  risk  stated  in  Decision  Coordinating  Paper  (or  PMD)  # (date).” 

II.  SYSTEM  PARAMETERS;  Describe  those  key  system  characteristics  and  performance  parameters 
which  will  be  tested  and  evaluated.  Do  not  attempt  to  list  all  parameters  that  will  be  measured  in  the 
test  program. 

III.  THE  CRITICAL  QUESTIONS  AND  AREAS  OF  RISK:  (Both  development  and  operational)  to 
be  addressed  by  test  and  evaluation  may  be  included  here  if  not  already  specified  in  the  PMD. 

IV.  GENERAL  OBJECTIVES:  Brief  summary  statements  of  both  DT&E  and  OT&E  objectives  n/ai/ 
be  provided  here.  Include  objectives  to  be  addressed  both  by  contractor-conducted  and  Air  Force- 
conducted  tests. 

V.  SPECIFIC  OBJECTIVES:  List  the  specific  objectives  for  DT&E  and  OT&E  which  address  those 
critical  issues  and  areas  of  risk  stated  in  the  DCP  and  PMD  which  can  be  resolved  by  testing.  Clearly 
indicate  the  relationship  between  the  issues/risks  and  the  test  objectives.  (Do  not  enumerate  test 
methods  or  procedures,  these  will  be  established  in  the  TEMP.)  Some  DT&E  and/or  OT&E  not  critical  to 
the  production  decision  may  occur  after  the  first  major  production  decision.  Therefore,  indicate  (either 
by  asterisk,  or  if  required  for  clarity,  by  sectioning)  which  objectives  must  be  satisfied  prior  to  the 
production  decision.  Incases  where  several  production  decisions  are  required,  e.g.,  (DSARC  II I A,  B,  or 
C)  indicate  which  objectives  are  to  be  satisfied  prior  to  each  decision. 
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Organization  and  Mission — Field 
AIR  FORCE  TEST  AND  EVALUATION  CENTER  (AFTEC) 

This  regulation  establishes  AFTEC  as  a separate  operating  agency  (SOA)  and  explains  its  mission, 
organization,  relationships,  policies  and  responsibilities. 


1.  Mission.  The  mission  of  AFTEC  is  to  manage 
the  Air  Force’s  Operational  Test  and  Evaluation 
(OT&E)  program  in  accordance  with  Air  Force 
policy.  AF’TEC  operates  within  the  Air  Force 
concept  of  T&E  as  outlined  in  AFR  80-14  and 
serves  as  the  principal  field  command  for  provid- 
ing OT&E  information  to  the  Air  Force  Chief  of 
Staff  (CSAF)  and  Secretary  of  the  Air  Force 
(SAF).  AFTEC  plans,  directs,  controls,  and  in- 
dependently evaluates  and  reports  on  OT&E  as 
well  as  recommends  OT&E  policy  to  HQ  USAF. 

2.  Organization.  Principal  AFTEC  organiza- 
tional elements  include: 

a.  AP’TEC-assigned  personnel  consisting  of 
test  directors,  managers,  monitors,  and  other 
specialists. 

b.  OT&E  teams  consisting  of  AFTEC-assigned 
personnel  and  other  resources  (for  example,  per- 
sonnel, forces,  and  ecjuipment)  provided  by  major 
commands  (MA.ICOMs)  according  to  HQ  USAF 
program  management  directives  (PMDs)  or  test 
directives  (TDs). 

8.  Command  Relationships: 

a.  The  AFTEC  commander  commands  a SOA 
and  reports  directly  to  the  CSAF. 

b.  MAJCOMs  support  AFTEC  as  directed  by 
HQ  USAF'  PMDs,  TDs,  and  other  directives. 

c.  AF’TEC  obtains  support  from  Air  Force, 
DOD,  and  governmental  ranges  or  centers  in  the 
same  manner  as  other  governmental  users  of  such 
facilities. 

d.  The  AFTFIC  commander  has  command 
jurisdiction  over  those  personnel,  facilities,  prop- 
erty, and  funds  organic  to  AF’TEC.  He  also  exer- 
cises operational  control  over  OT&E  team  re- 
sources during  specific  OT&E  programs.  How- 
ever, commanders  of  MAJCOMs  retain  command 
jurisdiction  over  those  resources  provided 
AF’TEC  OT&E  teams,  except  as  specified  in 
paragraph  4g  for  key  OT&E  team  personnel. 


Supersedes  AF’R  28-36,  1 January  1974.  (F’or 
summary  of  revised,  deleted,  or  added  material, 
see  signature  page.) 

OPR:  XOOE 
DISTRIBUTION:  F 


4.  Policies.  The  following  policies  apply  in  the 
management  of  the  Air  Force  OT&E  program. 

a.  OT&E  is  conducted  to  estimate  the  military 
utility,  to  include  the  operational  effectiveness, 
operational  suitability  and  operating  and  support 
cost  information  (as  defined  in  AFR  80-14)  of  a 
system,  subsystem,  or  equipment  item  (here- 
inafter referred  to  as  a system)  and  to  identify 
any  operational  deficiencies  of  the  .system. 

b.  OT&E  in  the  system  acquisition  process  is 
normally  conducted  in  phases: 

(1)  Initial  OT  & E (lOT&E).  While  all  OT&E 
objectives  should  be  thoroughly  examined  during 
lOT&E,  its  primary  objective  is  to  estimate  the 
military  utility  (as  well  as  identify  operational  de- 
ficiencies) of  the  system.  Since  such  an  estimate  is 
a major  factor  in  the  first  major  production  deci- 
sion, it  should  be  thorough  enough  to  provide  a 
reasonable  assessment  of  the  system's  military 
utility  in  its  intended  operational  environment. 
lOT&Fl  begins  as  early  as  possible  in  a system's 
life  cycle  and  ends  with  the  first  major  production 
decision.  However,  when  any  future  system  appli- 
cation, threat,  tactic,  or  doctrine  requires  a major 
modification  to  the  system,  the  modification  pro- 
gram itself  may  be  con.sidered  as  new  system  de- 
velopment and  require  lOT&E  leading  to  a pi'o- 
duction  decision. 

(2)  Follow-on  OT&E  (F’OT&E).  The  objec- 
tive of  this  phase  is  to  refine  the  estimates  of  a 
system’s  military  utility  as  the  basis  for  further 
possible  production  decisions,  system  configura- 
tion changes,  and  operational  use.  FOT&E  begins 
after  the  first  major  production  decision  and  may 
continue  throughout  the  system’s  life  cycle.  Pilot 
or  preproduction  test  items  (if  they  are  reasona- 
bly representative  of  the  expected  production 
items)  will  be  used  should  conditions  warrant 
continuing  OT&E  before  production  items  are 
available.  The  principal  AFTEC  role  in  F’OT&E 
is  that  of  furnishing  the  degree  of  independent 
operational  testing  and  evaluation  required  to  re- 
fine initial  estimates  made  in  lOT&E.  These  re- 
fined estimates  are  essential  to  support  further 
production  decisions  and  system  configuration 
changes  after  the  first  major  production  decision. 
The  extent  (phasing  and  activity  level)  of  the 
AF’TEC  role  in  F’OT&E  projects  is  determined  by 
HQ  USAF. 
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c.  OTX-F'  is  conducted  and  reported  in  conso- 
nance with  the  system's  opei'ational  emi)loyment 
and  maintenance  concepts  pi-ei)ared  by  MAJ- 
COMs  and  other  agencies  and  approved  bv  HQ 
USAF. 

d.  AFTEC  conducts  OT&K  on  all  major  and 
designated  non-major  Air  Force  systems.  Major 
systems  are  those  designated  by  the  Secretary  of 
Defense  (SFCDEF)  for  his  decision  to  initiate  or 
increase  program  commitments.  Designated 
non-major  systems  are  usually  identified  by  the 
HQ  USAF  for  si)ecial  attention,  but  AFTEC  may 
also  recommend  those  non-major  programs  which 
.AFTEC  should  conduct.  HQ  USAF  assigns 
OTi'iFl  lu'ograms  that  AFTEC  conducts  as  well  as 
some  OTii'E  programs  that  MA.JCOMs  conduct. 
Other  OT&E  programs  are  conducted  by  the 
•M.A.FCOMs  when  directed  by  their  commanders. 

e.  .AFTEC's  responsibilities  do  not  include  op- 
ei'ational training,  employment  exercises,  tests 
undertaken  primarily  to  develop  or  refine  tactics, 
techniiiues,  iirocedures,  and  doctrine  or  to  estab- 
lish or  verify  system  reliability  and  accuracy  fac- 
tors for  deployed  strategic  forces  as  recjuired  by 
the  Joint  Chiefs  of  Staff  for  the  Single  Integrated 
Olierational  Flan.  Information  derived  from 
OTA’E  may  be  used  by  the  MAJCOMs  to  refine 
tactics. 

f.  In  addition  to  its  own  system  acciuisition  pro- 
grams. the  Air  Force  conducts  joint  OT&E  of 
systems  (including  DDK&E-sponsored  joint 
(JT&E)  with  other  Armed  Services  and  gov- 
ernmental agencies,  as  directed  by  HQ  USAF. 
AFTEC  manages  the  Air  Force's  portion  of  this 
joint  OTA’E,  using  organic  capabilities  and  other 
resources  provided  by  MAJCOMs  according  to 
USAh'  directives.  The  AF'TEC  commander  estab- 
lishes liaison  with  the  other  services  and  gov- 
ernmental agencies  as  he  deems  appropriate  to 
conduct  designated  joint  OT&E  programs. 

g.  On  AFTEC-conducted  OT&E  programs,  the 
AFTEC  commander  designates  a test  director  to 
head  OT&E  teams.  The  AFTEC  commander  ap- 
points test  directors  from  (1)  AFTEC  personnel 
or  (2)  MAJCOMs  with  the  concurrence  of  the 
•M.AJCO.M  commander.  The  AFTEC  test  director 
exercises  operational  control  over  OT&E  team 
resources  during  a specific  OT&E  program  and 
may  exercise  authority  to  rate  the  performance  of 
key  .MAJCOM  OT&E  test  team  personnel  with 
agreement  of  the  affected  .MAJCOM. 

h.  While  OT&E  must  ensure  an  independent 
assessment  of  a system’s  military  utility,  opera- 
tional testing  is  often  conducted  concurrently 
with  development  testing.  At  such  times,  the 
planning  for  OT&E  and  DT&E  must  consider  in- 
tegration into  a combined  test  program.  In  this 
situation,  the  implementing  command  (normally 
AFSC)  provides  a Test  Director  and  has  respon- 


sibility for  DT&E  and  for  integrating  the  com- 
bined test  events. 

(1)  On  AFTPX'-conducted  OT&E  programs, 
the  AP’TEC  commander  assigns  a Test  Director 
for  OT&E  who  manages  the  OT&E  i)art  of  the 
combined  test  and  has  oi)erational  control  over 
the  OT&E  team  and  the  OT&E  ixu'tion  of  the  test 
program. 

(2)  The  resources  under  the  opei'ational  con- 
trol of  the  AP'TEC  OT&E  Test  Director  will,  in 
addition  to  supporting  the  OT&E  effort,  be  made 
available  as  specified  in  the  combined 
develo])ment/operational  test  plan,  to  support  the 
DT&E  portion  of  the  combined  test  program. 

i'.i)  .Additional  DT&E  support  from  the 
OT&E  test  team  is  made  available,  as  reiiuired, 
by  mutual  agreement  of  the  DT&tl  and  OT&E 
test  directors. 

(4)  .AFTflC  prepares  the  OT&E  jiart  of  the 
combined  plan,  coordinates  on  the  entire  plan, 
and  ensures  that  (4T&E  does  not  unnecessarily 
duplicate  any  lu'ogrammed  DT&Fl. 

i.  When  combined  testing  does  not  permit 
.AP'TEC  to  properly  assess  the  o()erational 
capabilities  of  the  system.  .AP’TEC  may  conduct 
some  separate  test  events  to  ensure  an  independ- 
ent operational  assessment.  These  separate 
OT&E  reipiirements  are  defined  as  an  integral 
jiart  of  the  combined  test.  The  .AP'TEC  com- 
mander has  full  responsibility  for  managing  and 
funding  such  activities. 

j.  For  HQ  USAF-designated,  MAJCO.M- 
conducted  OT&E  programs,  .AP’TFIC  will  partici- 
pate sufficiently  in  test  planning  to  permit  ap- 
proval of  the  test  plans  and  comment  on  the  final 
report  if  appropriate.  For  lOT&E,  .AP’TEC  will 
participate  with  sufficient  activity  and  program 
involvement  to  permit  a thorough  evaluation  of 
the  program. 

k.  To  assure  adequate  familiarity  with  systems 
that  are  expected  to  reach  OT&E,  AFTEC  will 
participate  in,  or  observe,  the  pre-DT&E  de- 
velopment of  major  and  other  HQ  USAF- 
designated  systems  to  the  extent  agreed  upon  by 
AF’TEC  and  the  implementing  command,  or  as 
directed  by  HQ  USAF.  This  may  include  locating 
AFTEC  team  personnel  at  the  .system  program 
offices  (SPOs),  or  with  SPO  personnel  at  contrac- 
tor facilities. 

5.  Responsibilities: 

a.  AFTEC  provides  OT&E  information  to  the 
SAF  and  the  CSAF’  in  preparation  for  DSARC 
and  Air  Force  decision  milestone  reviews  for 
major  programs,  and  as  otherwise  requested  by 
HQ  USAF.  The  Commander  of  AFTFIC  may  also 
request  attendance  at  specific  program  reviews. 
Such  attendance  is  contingent  ui)on  approval  by 
HQ  USAF. 
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h.  AFTKC  providt's  OTitK  informatiDii  to  sup- 
port th(jso  Air  Force  |)rocurcmc;it  re(|Uests  foi’ 
which  OTi'cK  information  is  statutorily  reciuircd 
to  ho  suj)pliod  to  C’onjii'oss. 

c.  AFTFC  rocoinmonds  to  fSAF  the  accom- 
plishmont  of  adoquato  OTA-F,. 

d.  AFTFC’  maintains  a data  file  on  all  Air  Foi’ce 
OTAK  |)ro^n•ams  which  includes  relevant  infor- 
mation from  system  DTAF,  OTAK.  joint  tests. 
\Vea])on  S.vstem  Fvaluation  [’i-ojii-ams.  and  othei’ 
appropriate  test  and  evaluation  pi'ojrrams  foi-  .\ii' 
Force  and  1)01)  evaluation. 

e.  AFTFC  assists  HC^  I'S.AF  in  the  i)re|)aration 
of  the  annual  I’rojrram  Objective  .Memoi'andum 
(I’O.M)  and  budfret  by  maintaininf;  and  updatiny^ 
annually  the  following  documents: 

(1)  The  OTA'F  .Master  l'ro>;ram  which  iden- 
tifies systems  to  be  tested,  the  scoi)e  and  objec- 
tives of  OTAF.  milestones.  i)articipatinfr  com- 
mands. and  estimates  of  i-esources  re(|uii’ed  for 
the  i)eriod  of  the  Five-Year  Defense  Program. 

(2)  The  OTA’F  .Master  Instrumentation/ 
Capabilities  Plan/Cuide  w hich  identifies  existing 
and  ret|uired  tfunded  and  unfunded)  .Air  Force 
OTA’F  instrumentation  and  facilities. 

f.  .AFTFC  prei)ares.  for  HQ  CS.AF  u.se  and  ap- 
)>roval.  the  OTA  F.  i)ortions  of  Decision  Coordinat- 
ing Pai)ers  (DCPs),  Pi-ogram  Memoranda.  F.MDs. 
Area  Coordinating  Papers,  and  similar  DOD 
documeiitation  including  critical  (piestions  and 
areas  of  risk  to  be  addressed  in  OTAFT 

g.  The  .AFTF’.C  commander  is  responsible  for 
OTAF  funding  which  includes: 

(1)  F'unds  "for  .AF'TFC  sui)poi’t  (overhead 
costs)."  These  costs  include  such  items  as  i)ay  for 
civilian  personnel,  as  well  as  travel,  transporta- 
tion. per  diem,  rents,  “facility  projects  by  con- 
tract." contractual  services,  supplies,  and  eipiip- 
ment  necessary  to  accomplish  .AF'TF’.C’s  mission. 

(2)  F’unds  (PAD)  to  support  speciHc  I()TAFi 
|)rograms.  both  AF’TFX’-conducted  and  other 
.M.A.ICD.M-conducted  (per  AF’M  172-1.  Vol  I). 

(3)  F’umls  (t)A.M)  to  support  s])ecific  F’DTAFi 
l)rograms  and  OTAFi  for  one-of-a-kind  systems 
such  as  simulators  and  Class  IV  A V modification 
programs  which  HQ  USAF’  has  directed  .AF'TFIC 
to  conduct  (per  AF’.M  172-1,  Vol  1). 

NOTFl:  Procurement  of  actual  test  items,  flying- 
hour  costs  of  supporting  aircraft,  munitions,  and 
regular  i)ay  of  personnel  are  not  included  in  AF’- 
TFlC’s  funding  responsibilities  except  when  re- 
i|uired  bv  the  DOD  uniform  funding  policy  for 
TAF. 

h.  On  AF’TF^C-conducted  OTAF  programs: 

(1)  The  AF’TFIC  test  director  assigns  specific 
tasks  to  OTAF;  team  members  and  directs  and 
controls  test  operations. 

(2)  The  OTAF  teams  plan,  conduct,  evaluate, 
and  report  on  assigned  OTAF]  programs.  Reports 


are  submitted  by  the  .AF’TFiC  commander  di- 
rectly the  C.'^.AF’  with  copies  provided  to  com- 
manders of  ai>proi)riate  .M.A-H’O.Ms. 

(3)  .AF’TFiC  participates  with  the  imi)lement- 
ing  command  (normally  .AF'SC)  in  prei)aring  the 
TAFi  Master  Plan  (TF].MP)/Test  and  Fvaluation 
Objectives  .Annex  (TFiO.A)  and  i)re|)ares  the 
OTAF]  jioi’tion  of  that  TFi.MP  TFiO.A  and  coordi- 
nates on  the  TFiMP  TFiO.A. 

i.  On  HQ  I’.S.A  F’-designated . .M.A.K’O.M- 
conducted  OTAF]  programs: 

(1)  .AF’TF]t’  must  api>rove  all  test  plans  be- 
fore they  are  im])lemented  (except  as  stated  in 
paragraph  4e).  To  facilitate  a|)proval  of  the  M.AJ- 
(’O.M  test  plan.  .AF’TF](’  particii)ates.  in  formulat- 
ing all  .M.A.K’OM  documentation,  test  oi’ders.  and 
so  forth,  associated  with  lOTAF]  and  ai)i)roi)i’iate 
F’OTAF]  ])i’ograms  (see  4b(2)  ).  .Additionally, 
under  this  category,  if  an  operational  test  pro- 
gram is  combined  with  a development  test  pro- 
gram. .AF’TFiC’  ap|)roves  the  lOTAF]  jun-tion  of 
and  coordinates  on  the  combined  test  program 
documentation. 

(2)  Re])orts  of  these  tests  are  submitted  to 
the  CS.AF’  through  the  AF'TFiC  commander  for 
evaluation  and  comment  on  lOTAF]  and  other  aj)- 
propriate  (')TAF]  programs.  Copies  of  the  .AF’TFiC 
commander’s  evaluation  and  comments  are  pro- 
vided to  commanders  of  appropriate  .M.AJCO.Ms. 
the  HQ  USAF'  OI’R  for  OTAF].  and  other  in- 
terested agencies. 

j.  On  M.AJC’OM-initiated  OTAF]  programs: 
.AF’TF]C  comments,  as  api)roi)riate,  on  test  plans 
and  evaluation  reports  and  includes  appropriate 
information  in  the  OTAF]  data  file  (reference 
paragraph  .od). 

h.  T&F]  Waivers.  In  the  case  of  major  programs, 
any  waiver  of  the  TAF]  outlined  in  the  approved 
DCP  is  granted  only  by  the  .Secretary  of  Defense. 
F’or  itrograms  directed  by  a program  memoran- 
dum (P.M),  waiver  of  TAF  outlined  in  the  P.M  ma.v 
be  approved  by  the  Secretary  of  the  .Air  F’orce  or 
a designated  Assistant  Secretary.  F'or  other  pro- 
grams, waiver  of  HQ  US.AF-a))proved  TAF  ob- 
jectives may  be  approved  by  HQ  USAF.  Ap- 
proved waivers  are  documented  in  the  appro- 
priate program  directives. 

7.  Rep»»rts.  Reports  published  by  AF’TF]C  will  be 
submitted  bv  AF’TF]C/CC  to  the  participating 
MA.JCOMs,  to  the  HQ  USAF  OPR  for  OTAF,  to 
other  interested  agencies,  and  to  CSAF’  when  ap- 
propriate. Categories  of  reports  published  by 
AF’TFIC  are; 

a.  Status  Reports.  These  reports  provide 
periodic  (weekly,  monthly,  etc)  information  or 
important  test  findings. 
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b.  Quarterly  Report.  This  report,  normally  a 
letter  report,  will  provide  information  on  appro- 
priate major  and  designated  non-major  programs 
for  senior  management  level  review. 

e.  Interim  Report.  This  report  is  written  at  the 
request  of  HQ  USAF  when  recjuired  for  a major 
program  milestone  or  if  operational  test  findings 
result  in  a significant  impact  on  the  test  pi'ogram. 


fl.  Final  Report.  This  report  jtresents  the  final 
test  results. 

8.  Direct  Communication.  Direct  communica- 
tion is  authorized  between  the  Commander,  AF- 
TFIC,  and  the  commanders  of  other  USAF  com- 
mands, other  government  agencies,  the  Office  of 
the  SEICDEF,  and  other  Armed  Services  on  mat- 
ters pertaining  to  AFTEC  resi)onsibilities. 


BY  ORDER  OF  THE  SECRETARY  OF  THE  AIR  FORCE 


OFFICIAL  DAVID  C.  .lONES,  General,  USAF 

Chief  of  Staff 

•JA.MES  J.  SHEPARD,  Colonel,  USAF 
Director  of  Administration 

SUMMARY  OF  REVISED,  DELETED,  OR  ADDED  MATERIAL 

This  revision  clarifies  management  relationships  and  responsibilities  between  AFTEC,  the  imi)lement- 
ing  command,  and  other  commands  in  the  conduct  of  OT&E  and  combined  development/operational 
testing  (para  2,  8,  4,  5,  & 8);  stipulates  the  participation,  early  in  system  development  of  OT&E  per- 
•sonnel  (para  4j);  redefines  lOT&E  as  that  OT&E  conducted  prior  to  the  first  major  production  decision, 
and  F'OT&El  as  all  OT&EI  after  that  decision  (para  4b(l)  and  (2)  );  addresses  the  conduct  of  ODDR&E 
sponsored  joint  Service  testing  (para  4f);  outlines  the  revised  OT&E  funding  policy  ai)plicable  to 
.AE'TEX  (para  .5g);  reciuires  AFTEC  to  provide  information  to  SAF  and  CSAF  in  ])rei)aration  for  deci- 
sion milestone  reviews  for  major  programs  (para  ba);  limits  AETEC  reviews  of  MAJCOM  conducted 
programs  to  IOT&E2  and  selected  E’OT&Ei  (para  4b  and  j);  and  changes  the  title  of  the  OT&El  .Master 
Instrumentation/Capabilities  Program  to  the  OT&Ei  .Master  Instrumentation/Ca])abilities  Guide/Plan 
(para  5e(2)  );  defines  the  authority,  distribution,  and  categories  of  AE'TEiC  OT&E]  rei)orts  (para  7). 
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